Oct 30 19:20:05.403661 ip-10-0-140-248 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:20:05.778402 ip-10-0-140-248 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:20:05.778402 ip-10-0-140-248 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:20:05.778402 ip-10-0-140-248 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:20:05.778402 ip-10-0-140-248 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:20:05.782916 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.782823    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:20:05.782916 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.782913    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:20:05.782916 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.782919    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.782925    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.782930    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.782940    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.782944    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.782948    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.782954    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.782958    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.782962    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.782967    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.782972    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.782976    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.782981    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.782989    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.782997    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783024    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783045    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783052    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783057    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:20:05.783092 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783062    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783068    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783073    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783078    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783083    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783088    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783092    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783097    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783101    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783113    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783121    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783131    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783137    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783143    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783149    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783154    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783159    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783164    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783168    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:20:05.783973 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783173    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783178    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783188    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783193    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783200    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783205    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783210    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783214    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783219    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783224    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783228    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783233    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783237    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783241    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783246    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783261    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783266    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783270    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783275    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:20:05.784631 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783280    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783285    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783290    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783294    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783299    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783303    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783308    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783317    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783322    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783327    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783331    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783338    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783343    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783347    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783352    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783356    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783377    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783382    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783387    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783391    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:20:05.785084 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783399    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783404    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783409    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783413    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783417    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783422    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783426    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783430    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783435    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783440    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783444    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783448    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783453    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783461    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783466    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783470    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783474    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783479    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783483    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:20:05.785578 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783487    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783495    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783499    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783504    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783508    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783513    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783521    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.783525    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785210    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785219    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785223    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785226    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785229    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785233    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785236    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785238    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785241    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785244    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785246    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785249    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:20:05.786038 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785252    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785254    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785257    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785259    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785262    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785264    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785271    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785274    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785277    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785279    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785282    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785285    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785288    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785290    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785293    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785296    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785298    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785301    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785303    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785306    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:20:05.786541 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785308    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785312    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785315    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785318    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785321    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785324    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785326    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785329    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785331    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785334    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785336    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785339    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785342    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785344    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785348    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785352    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785356    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785375    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785378    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:20:05.787011 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785380    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785383    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785386    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785388    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785391    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785394    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785396    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785399    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785402    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785405    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785407    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785410    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785412    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785415    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785418    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785421    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785423    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785426    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785428    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785439    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:20:05.787472 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785442    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785445    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785447    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785450    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785452    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785455    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785458    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785461    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785463    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785466    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785468    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785471    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785474    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785476    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785479    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785481    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785484    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785486    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785489    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:20:05.787933 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785491    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785494    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785497    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785500    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785502    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785505    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785508    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785511    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785514    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785516    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785519    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785524    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785529    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785532    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785534    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.785537    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786059    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786066    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786070    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:20:05.788399 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786073    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786076    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786078    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786081    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786084    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786086    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786089    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786091    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786094    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786098    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786101    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786103    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786106    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786109    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786112    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786115    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786118    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786120    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786123    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786126    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:20:05.788857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786128    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786131    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786133    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786136    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786138    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786141    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786143    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786146    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786148    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786151    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786153    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786156    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786159    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786161    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786164    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786166    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786169    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786171    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786174    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:20:05.789343 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786176    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786179    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786181    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786188    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786192    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786196    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786199    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786201    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786204    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786207    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786209    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786212    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786214    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786217    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786219    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786222    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786224    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786227    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786229    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:20:05.789809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786232    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786234    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786237    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786239    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786242    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786244    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786247    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786249    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786252    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786255    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786257    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786260    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786262    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786265    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786267    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786271    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786276    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786279    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786283    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:20:05.790263 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786286    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786289    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786292    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786295    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786297    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786301    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786303    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786306    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786308    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786311    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786313    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786316    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786318    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786321    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786323    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786326    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786328    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786331    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786335    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786337    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:20:05.790760 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786340    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786343    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786345    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786348    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786350    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.786353    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786459    2582 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786468    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786476    2582 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786485    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786491    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786498    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786505    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786510    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786513    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786516    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786519    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786522    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786526    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786529    2582 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786532    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786535    2582 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786538    2582 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786541    2582 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:20:05.791222 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786546    2582 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786549    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786552    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786555    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786558    2582 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786562    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786565    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786568    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786571    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786574    2582 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786579    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786581    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786585    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786588    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786592    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786595    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786598    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786602    2582 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786605    2582 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786610    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786614    2582 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786617    2582 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786620    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786623    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:20:05.791782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786626    2582 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786630    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786633    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786636    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786639    2582 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786642    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786645    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786648    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786651    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786654    2582 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786657    2582 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786659    2582 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786663    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786666    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786669    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786673    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786676    2582 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786679    2582 flags.go:64] FLAG: --help="false"
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786681    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-140-248.ec2.internal"
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786685    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786688    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786691    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786694    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:20:05.792415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786697    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786700    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786704    2582 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786707    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786711    2582 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786714    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786718    2582 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786720    2582 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786723    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786726    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786729    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786732    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786735    2582 flags.go:64] FLAG: --lock-file=""
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786755    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786759    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786763    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786768    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786772    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786775    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786778    2582 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786781    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786784    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786787    2582 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786789    2582 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786794    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:20:05.793041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786797    2582 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786801    2582 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786804    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786807    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786811    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786814    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786817    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786819    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786822    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786832    2582 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786835    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786839    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786842    2582 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786845    2582 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786848    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786851    2582 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786854    2582 flags.go:64] FLAG: --port="10250"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786857    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786859    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-057cf47e1ce072bcf"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786862    2582 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786866    2582 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786868    2582 flags.go:64] FLAG: --register-node="true"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786871    2582 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786880    2582 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786883    2582 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786885    2582 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:20:05.793652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786889    2582 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786892    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786895    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786899    2582 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786901    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786904    2582 flags.go:64] FLAG: --runonce="false"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786907    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786910    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786913    2582 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786916    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786919    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786922    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786925    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786928    2582 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786931    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786935    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786938    2582 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786942    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786945    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786948    2582 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786951    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786956    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786959    2582 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786962    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786966    2582 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:20:05.794272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786969    2582 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786972    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786975    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786978    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786981    2582 flags.go:64] FLAG: --v="2"
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786985    2582 flags.go:64] FLAG: --version="false"
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786989    2582 flags.go:64] FLAG: --vmodule=""
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786993    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.786996    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787130    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787134    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787137    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787140    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787143    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787145    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787148    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787150    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787153    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787156    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787159    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787162    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787164    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:20:05.794946 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787168    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787171    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787175    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787178    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787181    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787183    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787186    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787188    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787191    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787194    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787197    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787199    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787202    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787204    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787207    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787210    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787212    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787215    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787217    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:20:05.795480 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787220    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787223    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787225    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787229    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787233    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787236    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787239    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787242    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787245    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787247    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787250    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787253    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787256    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787260    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787263    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787266    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787269    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787272    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787274    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:20:05.795963 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787277    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787279    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787282    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787284    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787287    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787290    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787292    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787295    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787297    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787300    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787303    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787305    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787308    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787310    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787313    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787315    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787318    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787321    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787323    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:20:05.796426 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787326    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787328    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787331    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787333    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787336    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787338    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787341    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787345    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787348    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787352    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787354    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787371    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787375    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787378    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787381    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787385    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787388    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787390    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787393    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:20:05.796869 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787396    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:20:05.797317 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787399    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:20:05.797317 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787401    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:20:05.797317 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787403    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:20:05.797317 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787406    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:20:05.797317 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787409    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:20:05.797317 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787411    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:20:05.797317 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787414    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:20:05.797317 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787417    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:20:05.797317 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787419    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:20:05.797317 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787422    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:20:05.797317 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787424    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:20:05.797317 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787427    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:20:05.797317 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787430    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:20:05.797317 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787433    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:20:05.797317 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787435    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:20:05.797317 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787438    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:20:05.797711 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.787464    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:20:05.797711 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787503    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:20:05.797711 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787508    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:20:05.797711 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787511    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:20:05.797711 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787515    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:20:05.797711 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787519    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:20:05.797711 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787522    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:20:05.797711 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787524    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:20:05.797711 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787527    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:20:05.797711 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787529    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:20:05.797711 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787532    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:20:05.797711 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787534    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:20:05.797711 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787537    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:20:05.797711 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787539    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:20:05.797711 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787542    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:20:05.797711 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787545    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787547    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787550    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787552    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787555    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787557    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787560    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787562    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787565    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787567    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787570    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787572    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787575    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787578    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787580    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787583    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787585    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787588    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787590    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787593    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:20:05.798145 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787595    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787598    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787602    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787605    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787607    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787610    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787612    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787615    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787617    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787620    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787622    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787625    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787628    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787631    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787633    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787635    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787639    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787641    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787644    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:20:05.798664 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787646    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787649    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787651    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787654    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787659    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787662    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787665    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787668    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787670    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787673    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787676    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787678    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787681    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787683    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787686    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787690    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787693    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787695    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787698    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:20:05.799192 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787700    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787703    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787705    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787708    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787710    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787713    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787716    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787718    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787721    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787723    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787726    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787728    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787731    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787733    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787736    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787738    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787741    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787743    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787746    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:20:05.799678 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787748    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787751    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787753    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787756    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787758    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787760    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787763    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787765    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787768    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787772    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787775    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787777    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787782    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787786    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.787788    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.794215    2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.794228    2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.794241    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.794246    2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:20:05.800177 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794299    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794304    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794308    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794311    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794314    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794317    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794320    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794323    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794326    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794328    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794331    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794333    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794336    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794339    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794341    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794344    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794347    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794349    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794352    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794355    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:20:05.800704 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794373    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794378    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794382    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794386    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794399    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794402    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794405    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794408    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794411    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794420    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794423    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794426    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794429    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794432    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794434    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794437    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794440    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794442    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794445    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:20:05.801471 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794448    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794450    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794453    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794455    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794458    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794460    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794463    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794465    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794468    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794470    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794473    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794476    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794478    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794481    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794483    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794486    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794489    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794491    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794494    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:20:05.801921 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794496    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794499    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794501    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794505    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794510    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794513    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794516    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794518    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794521    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794523    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794526    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794528    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794531    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794534    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794536    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794539    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794541    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794544    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794547    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794549    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:20:05.802668 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794552    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794555    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794557    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794560    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794562    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794565    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794567    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794570    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794573    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794575    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794578    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794580    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794583    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794585    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794588    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794593    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794597    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794605    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794608    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794611    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:20:05.803161 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794615    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:20:05.803706 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794618    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:20:05.803706 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794621    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:20:05.803706 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794624    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:20:05.803706 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794626    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:20:05.803706 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794629    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:20:05.803706 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794633    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:20:05.803706 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794636    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:20:05.803706 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.794665    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:20:05.803706 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794799    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:20:05.803706 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794804    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:20:05.803706 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794807    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:20:05.803706 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794810    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:20:05.803706 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794814    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:20:05.803706 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794818    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:20:05.803706 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794821    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794824    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794826    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794829    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794831    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794834    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794836    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794839    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794841    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794844    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794846    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794849    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794852    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794855    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794858    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794861    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794863    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794866    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794869    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794871    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:20:05.804060 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794874    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794876    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794879    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794881    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794884    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794887    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794889    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794892    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794894    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794897    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794900    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794903    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794905    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794908    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794911    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794913    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794916    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794919    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794921    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:20:05.804785 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794924    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794927    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794929    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794932    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794934    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794937    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794940    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794942    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794945    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794948    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794951    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794954    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794957    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794960    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794962    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794965    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794968    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794970    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794973    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794975    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:20:05.805237 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794978    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794981    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794984    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794988    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794991    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794994    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.794997    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795000    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795002    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795005    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795007    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795010    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795012    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795015    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795017    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795020    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795022    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795025    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795028    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795030    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:20:05.805775 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795033    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795035    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795038    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795041    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795044    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795046    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795048    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795051    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795054    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795057    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795059    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795062    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795064    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795067    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795069    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795072    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795074    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795077    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795079    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795082    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:20:05.806282 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795084    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:20:05.806773 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.795107    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:20:05.806773 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795139    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:20:05.806773 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795143    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:20:05.806773 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795146    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:20:05.806773 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795149    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:20:05.806773 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795152    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:20:05.806773 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795154    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:20:05.806773 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795157    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:20:05.806773 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795159    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:20:05.806773 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795162    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:20:05.806773 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795165    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:20:05.806773 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795167    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:20:05.806773 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795170    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:20:05.806773 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795172    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:20:05.806773 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795175    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795178    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795180    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795183    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795186    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795188    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795191    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795193    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795196    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795199    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795202    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795204    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795207    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795209    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795212    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795214    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795217    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795219    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795222    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:20:05.807168 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795225    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795228    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795230    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795233    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795235    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795238    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795240    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795243    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795245    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795248    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795251    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795253    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795256    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795258    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795261    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795266    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795268    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795272    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795276    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795279    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:20:05.807757 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795282    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795285    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795287    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795290    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795293    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795295    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795298    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795300    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795303    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795306    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795308    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795312    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795314    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795317    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795319    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795322    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795325    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795328    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795330    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795333    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:20:05.808406 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795335    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795338    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795341    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795343    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795346    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795348    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795351    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795355    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795371    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795374    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795377    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795380    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795382    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795385    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795387    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795390    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795392    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795395    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795398    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:20:05.809155 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795402    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795406    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795409    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795412    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795416    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795419    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795422    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795424    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795427    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795430    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795432    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795435    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795437    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795440    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:05.795444    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.795492    2582 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.797637    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.798559    2582 server.go:1014] "Starting client certificate rotation"
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.798658    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:20:05.809809 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.798696    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:20:05.810257 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.800298    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:20:05.810257 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.802120    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:20:05.813661 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.813646    2582 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:20:05.816333 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.816321    2582 log.go:25] "Validated CRI v1 image API"
Oct 30 19:20:05.817091 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.817070    2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:20:05.823250 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.823231    2582 fs.go:135] Filesystem UUIDs: map[1dfc8d5e-3db9-4c70-9e2f-1ed6139f2942:/dev/nvme0n1p3 23ef4387-7852-4a0c-9768-67d330c12eb0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 19:20:05.823334 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.823249    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 19:20:05.828135 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.828030    2582 manager.go:217] Machine: {Timestamp:2026-10-30 19:20:05.826746952 +0000 UTC m=+0.311403055 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094397 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21513917c62edb1174617ecd4260ba SystemUUID:ec215139-17c6-2edb-1174-617ecd4260ba BootID:6cc37678-bf2e-461b-8323-6dbbffd96943 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d6:bd:6d:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d6:bd:6d:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:cb:28:0e:ee:f6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:20:05.828135 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.828134    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:20:05.828224 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.828205    2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:20:05.829180 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.829159    2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:20:05.829300 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.829182    2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-248.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:20:05.829340 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.829302    2582 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:20:05.829340 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.829309    2582 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:20:05.829404 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.829340    2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:20:05.829404 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.829355    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:20:05.830938 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.830928    2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:20:05.834272 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.834260    2582 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:20:05.834403 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.834394    2582 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:20:05.835059 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.835051    2582 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:20:05.835087 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.835063    2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:20:05.835087 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.835076    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:20:05.835087 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.835087    2582 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:20:05.835164 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.835096    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:20:05.835423 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.835411    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:20:05.837434 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.837416    2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:20:05.843165 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.843153    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:20:05.843872 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.843861    2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:20:05.843907 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.843890    2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:20:05.844041 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.844032    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:20:05.844070 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.844053    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:20:05.844070 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.844059    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:20:05.844070 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.844065    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:20:05.844070 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.844070    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:20:05.844162 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.844075    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:20:05.844621 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.844613    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:20:05.844657 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.844622    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:20:05.844657 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.844629    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:20:05.844657 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.844635    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:20:05.844657 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.844644    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:20:05.844657 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.844652    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:20:05.845587 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.845579    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:20:05.845625 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.845588    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:20:05.849220 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.849208    2582 server.go:1263] "Started kubelet"
Oct 30 19:20:05.849349 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.849285    2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:20:05.849402 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.849336    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:20:05.849440 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.849429    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:20:05.853060 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.853027    2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:20:05.853129 ip-10-0-140-248 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:20:05.857585 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.857460    2582 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:20:05.860347 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:05.860328    2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 19:20:05.862455 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.862439    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:20:05.862882 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.862867    2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:20:05.862960 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:05.862944    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-248.ec2.internal\" not found"
Oct 30 19:20:05.863605 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.863584    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:20:05.863605 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.863596    2582 factory.go:55] Registering systemd factory
Oct 30 19:20:05.863741 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.863611    2582 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:20:05.863741 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.863586    2582 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:20:05.863741 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.863659    2582 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:20:05.863741 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.863699    2582 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:20:05.863741 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.863708    2582 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:20:05.863887 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.863829    2582 factory.go:153] Registering CRI-O factory
Oct 30 19:20:05.863887 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.863843    2582 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:20:05.863956 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.863900    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:20:05.863956 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.863919    2582 factory.go:103] Registering Raw factory
Oct 30 19:20:05.863956 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.863935    2582 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:20:05.864801 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.864786    2582 manager.go:319] Starting recovery of all containers
Oct 30 19:20:05.874298 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.874281    2582 manager.go:324] Recovery completed
Oct 30 19:20:05.878156 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.878141    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:20:05.879902 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.879877    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:20:05.879975 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.879915    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:20:05.879975 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.879926    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:20:05.880477 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.880462    2582 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:20:05.880477 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.880475    2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:20:05.880585 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.880461    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:20:05.880585 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.880495    2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:20:05.881984 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.881960    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:20:05.882055 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.881991    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:20:05.882055 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.882007    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:20:05.882730 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:05.882701    2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 19:20:05.883996 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:05.882701    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-248.ec2.internal.18e364e3b9163abd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-248.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-248.ec2.internal,},FirstTimestamp:2026-10-30 19:20:05.849184957 +0000 UTC m=+0.333841059,LastTimestamp:2026-10-30 19:20:05.849184957 +0000 UTC m=+0.333841059,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-248.ec2.internal,}"
Oct 30 19:20:05.892779 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.892755    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-248.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 19:20:05.892983 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.892965    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:20:05.893055 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.893013    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8kvwt"
Oct 30 19:20:05.897663 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:05.897624    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 19:20:05.897761 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:05.897635    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-248.ec2.internal.18e364e3baeaefc4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-248.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-248.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-248.ec2.internal,},FirstTimestamp:2026-10-30 19:20:05.879902148 +0000 UTC m=+0.364558250,LastTimestamp:2026-10-30 19:20:05.879902148 +0000 UTC m=+0.364558250,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-248.ec2.internal,}"
Oct 30 19:20:05.915046 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.915030    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:20:05.943745 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:05.943727    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:20:05.963496 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:05.963469    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-248.ec2.internal\" not found"
Oct 30 19:20:06.020974 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.020956    2582 policy_none.go:50] "Start"
Oct 30 19:20:06.021049 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.020981    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:20:06.021049 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.020992    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:20:06.021922 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.021912    2582 policy_none.go:44] "Start"
Oct 30 19:20:06.065025 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.054829    2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:20:06.065025 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.057215    2582 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:20:06.065025 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:06.057244    2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:20:06.065025 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.057253    2582 server.go:87] "Starting device plugin registration server"
Oct 30 19:20:06.065025 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.057454    2582 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:20:06.065025 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.057464    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:20:06.065025 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.057638    2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:20:06.065025 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.057689    2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:20:06.065025 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.057694    2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:20:06.065025 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:06.058143    2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:20:06.065025 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:06.058179    2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-248.ec2.internal\" not found"
Oct 30 19:20:06.097233 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.097210    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:20:06.098356 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.098339    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:20:06.098949 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.098934    2582 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:20:06.099013 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.098972    2582 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:20:06.099056 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:06.099013    2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 19:20:06.112734 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:06.112719    2582 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-248.ec2.internal\" not found" node="ip-10-0-140-248.ec2.internal"
Oct 30 19:20:06.114332 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.114318    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:20:06.158432 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.158415    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:20:06.159197 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.159180    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:20:06.159282 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.159207    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:20:06.159282 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.159218    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:20:06.159282 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.159241    2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-248.ec2.internal"
Oct 30 19:20:06.180389 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.180354    2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-248.ec2.internal"
Oct 30 19:20:06.180515 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:06.180393    2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-248.ec2.internal\": node \"ip-10-0-140-248.ec2.internal\" not found"
Oct 30 19:20:06.199545 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.199519    2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal"]
Oct 30 19:20:06.199661 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.199565    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:20:06.202091 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.202075    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:20:06.202178 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.202104    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:20:06.202178 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.202117    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:20:06.204408 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.204393    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:20:06.204530 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.204515    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:06.204573 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.204543    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:20:06.205213 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.205196    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:20:06.205303 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.205222    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:20:06.205303 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.205235    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:20:06.205430 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.205309    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:20:06.205430 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.205331    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:20:06.205430 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.205345    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:20:06.207355 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.207339    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:06.207457 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.207382    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:20:06.208172 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.208156    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:20:06.208259 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.208182    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:20:06.208259 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.208193    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:20:06.219318 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:06.219303    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-248.ec2.internal\" not found"
Oct 30 19:20:06.232875 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:06.232858    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-248.ec2.internal\" not found" node="ip-10-0-140-248.ec2.internal"
Oct 30 19:20:06.235243 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:06.235229    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-248.ec2.internal\" not found" node="ip-10-0-140-248.ec2.internal"
Oct 30 19:20:06.265607 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.265588    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2670e93fcc64eb8a9f070c9bd973c8ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal\" (UID: \"2670e93fcc64eb8a9f070c9bd973c8ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:06.265681 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.265612    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2670e93fcc64eb8a9f070c9bd973c8ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal\" (UID: \"2670e93fcc64eb8a9f070c9bd973c8ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:06.265681 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.265627    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e9154b2294c90c44348210589c7863a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-248.ec2.internal\" (UID: \"4e9154b2294c90c44348210589c7863a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:06.319777 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:06.319731    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-248.ec2.internal\" not found"
Oct 30 19:20:06.366059 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.366040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2670e93fcc64eb8a9f070c9bd973c8ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal\" (UID: \"2670e93fcc64eb8a9f070c9bd973c8ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:06.366152 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.366062    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e9154b2294c90c44348210589c7863a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-248.ec2.internal\" (UID: \"4e9154b2294c90c44348210589c7863a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:06.366152 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.366077    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2670e93fcc64eb8a9f070c9bd973c8ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal\" (UID: \"2670e93fcc64eb8a9f070c9bd973c8ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:06.366152 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.366129    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2670e93fcc64eb8a9f070c9bd973c8ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal\" (UID: \"2670e93fcc64eb8a9f070c9bd973c8ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:06.366268 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.366149    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2670e93fcc64eb8a9f070c9bd973c8ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal\" (UID: \"2670e93fcc64eb8a9f070c9bd973c8ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:06.366268 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.366169    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e9154b2294c90c44348210589c7863a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-248.ec2.internal\" (UID: \"4e9154b2294c90c44348210589c7863a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:06.420646 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:06.420612    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-248.ec2.internal\" not found"
Oct 30 19:20:06.521263 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:06.521240    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-248.ec2.internal\" not found"
Oct 30 19:20:06.537420 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.537395    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:06.539536 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.539519    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:06.622204 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:06.622063    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-248.ec2.internal\" not found"
Oct 30 19:20:06.722486 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:06.722471    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-248.ec2.internal\" not found"
Oct 30 19:20:06.798907 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.798892    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 19:20:06.799287 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.799006    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:20:06.799287 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.799046    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:20:06.799287 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.799053    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:20:06.823185 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:06.823163    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-248.ec2.internal\" not found"
Oct 30 19:20:06.897567 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.897511    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:15:05 +0000 UTC" deadline="2028-04-05 17:23:08.782077818 +0000 UTC"
Oct 30 19:20:06.897567 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:06.897538    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12550h3m1.884543002s"
Oct 30 19:20:06.923589 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:06.923566    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-248.ec2.internal\" not found"
Oct 30 19:20:07.024415 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:07.024389    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-248.ec2.internal\" not found"
Oct 30 19:20:07.031533 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.031516    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:20:07.032796 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.032711    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:20:07.032893 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.032814    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:20:07.032893 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.032829    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:20:07.100589 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:07.100561    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9154b2294c90c44348210589c7863a.slice/crio-8bc6f2de22212ef491069993807ee1b09446c52f5dee4c880b8e8f28a5c4cb40 WatchSource:0}: Error finding container 8bc6f2de22212ef491069993807ee1b09446c52f5dee4c880b8e8f28a5c4cb40: Status 404 returned error can't find the container with id 8bc6f2de22212ef491069993807ee1b09446c52f5dee4c880b8e8f28a5c4cb40
Oct 30 19:20:07.104449 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.104437    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:20:07.124501 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:07.124483    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-248.ec2.internal\" not found"
Oct 30 19:20:07.224901 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:07.224861    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-248.ec2.internal\" not found"
Oct 30 19:20:07.283059 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.283042    2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:20:07.364148 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.364131    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:07.385327 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.385311    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:20:07.385424 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.385412    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:07.405942 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.404697    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:20:07.616857 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:07.616829    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2670e93fcc64eb8a9f070c9bd973c8ad.slice/crio-0008bcf05fe731a028bb8218b6d25cc4481c6f5e08353eed82e64ec4cf016693 WatchSource:0}: Error finding container 0008bcf05fe731a028bb8218b6d25cc4481c6f5e08353eed82e64ec4cf016693: Status 404 returned error can't find the container with id 0008bcf05fe731a028bb8218b6d25cc4481c6f5e08353eed82e64ec4cf016693
Oct 30 19:20:07.666818 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.666794    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:20:07.836065 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.836042    2582 apiserver.go:52] "Watching apiserver"
Oct 30 19:20:07.864049 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.863992    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 19:20:07.874717 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.874691    2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:20:07.875024 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.875007    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal","openshift-multus/multus-additional-cni-plugins-jjm2b","openshift-multus/network-metrics-daemon-vcr5t","openshift-cluster-node-tuning-operator/tuned-2gf9g","openshift-dns/node-resolver-n9frl","openshift-image-registry/node-ca-qrsz7","openshift-multus/multus-bw8v2","kube-system/global-pull-secret-syncer-27xsq","openshift-network-diagnostics/network-check-target-6dnzx","openshift-network-operator/iptables-alerter-lqm86","openshift-ovn-kubernetes/ovnkube-node-6rkvc","kube-system/konnectivity-agent-l65sf","kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"]
Oct 30 19:20:07.877426 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.877399    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:07.879667 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.879647    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.881731 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.881712    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:07.881814 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:07.881779    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcr5t" podUID="029fdc09-aaca-4dc2-90ff-86e05c26bf9e"
Oct 30 19:20:07.883888 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.883868    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:07.885967 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.885951    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n9frl"
Oct 30 19:20:07.888033 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.888017    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qrsz7"
Oct 30 19:20:07.890143 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.890124    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:07.890216 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:07.890189    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnzx" podUID="c1a81d1f-b828-4a0d-a57a-df3d493947d5"
Oct 30 19:20:07.891641 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.891624    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gx5x8\""
Oct 30 19:20:07.891727 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.891702    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:20:07.891783 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.891764    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:20:07.891884 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.891869    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:20:07.891968 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.891957    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:20:07.892941 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.892924    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:20:07.893028 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.892977    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:20:07.893099 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.893087    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:20:07.893267 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.893254    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:20:07.893341 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.893316    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8rhbj\""
Oct 30 19:20:07.893458 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.893447    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:20:07.893516 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.893506    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:07.895668 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.895481    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:07.897852 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.897831    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lqm86"
Oct 30 19:20:07.898735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.898718    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:20:07.898735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.898729    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:20:07.898883 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.898771    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:20:07.898883 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.898835    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hxbtq\""
Oct 30 19:20:07.900308 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.900289    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:07.900424 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:07.900391    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27xsq" podUID="bbd278f9-76f5-41ed-93d1-435804741834"
Oct 30 19:20:07.902865 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.902588    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l65sf"
Oct 30 19:20:07.909492 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.909474    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:20:07.909585 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.909572    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gzzkd"
Oct 30 19:20:07.964655 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.964637    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:20:07.976595 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.976573    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/954bf0fe-4287-4435-9abb-fd06dc478099-agent-certs\") pod \"konnectivity-agent-l65sf\" (UID: \"954bf0fe-4287-4435-9abb-fd06dc478099\") " pod="kube-system/konnectivity-agent-l65sf"
Oct 30 19:20:07.976684 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.976600    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-system-cni-dir\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:07.976684 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.976614    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-daemon-config\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:07.976684 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.976669    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-run-netns\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.976799 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.976697    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-node-log\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.976799 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.976717    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-socket-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:07.976799 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.976737    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aacc7bb9-0109-4984-884f-50f8d2144559-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:07.976799 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.976753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-lib-modules\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:07.976799 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.976789    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqxbc\" (UniqueName: \"kubernetes.io/projected/0b45e511-0890-4a70-96d7-01ba249d65cb-kube-api-access-lqxbc\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:07.976968 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.976817    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-kubelet\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.976968 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.976834    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-registration-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:07.976968 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.976875    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-sys-fs\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:07.976968 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.976902    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkgsr\" (UniqueName: \"kubernetes.io/projected/093cd549-ebec-45f5-b888-42019476c70c-kube-api-access-tkgsr\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:07.976968 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.976943    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aacc7bb9-0109-4984-884f-50f8d2144559-system-cni-dir\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:07.977123 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977001    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-modprobe-d\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:07.977123 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-run\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:07.977123 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977075    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f29fd70f-b10e-4154-8652-6b17fbe3cf13-iptables-alerter-script\") pod \"iptables-alerter-lqm86\" (UID: \"f29fd70f-b10e-4154-8652-6b17fbe3cf13\") " pod="openshift-network-operator/iptables-alerter-lqm86"
Oct 30 19:20:07.977123 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977113    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-run-netns\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:07.977262 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977134    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-run-ovn-kubernetes\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.977262 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977150    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8c299f86-7f6e-4666-ab9c-542537e6ff8e-env-overrides\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.977262 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977171    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90c7079a-e03a-4d3e-bb41-b61ce36292ec-hosts-file\") pod \"node-resolver-n9frl\" (UID: \"90c7079a-e03a-4d3e-bb41-b61ce36292ec\") " pod="openshift-dns/node-resolver-n9frl"
Oct 30 19:20:07.977262 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977189    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-sys\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:07.977439 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977261    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-run-multus-certs\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:07.977439 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977293    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/954bf0fe-4287-4435-9abb-fd06dc478099-konnectivity-ca\") pod \"konnectivity-agent-l65sf\" (UID: \"954bf0fe-4287-4435-9abb-fd06dc478099\") " pod="kube-system/konnectivity-agent-l65sf"
Oct 30 19:20:07.977439 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977319    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-run-k8s-cni-cncf-io\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:07.977439 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977339    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drlzm\" (UniqueName: \"kubernetes.io/projected/aacc7bb9-0109-4984-884f-50f8d2144559-kube-api-access-drlzm\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:07.977439 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977380    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-tuned\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:07.977439 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.977439 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977432    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c299f86-7f6e-4666-ab9c-542537e6ff8e-ovn-node-metrics-cert\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.977660 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977460    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5xl5\" (UniqueName: \"kubernetes.io/projected/8c299f86-7f6e-4666-ab9c-542537e6ff8e-kube-api-access-d5xl5\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.977660 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977481    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spzw5\" (UniqueName: \"kubernetes.io/projected/f29fd70f-b10e-4154-8652-6b17fbe3cf13-kube-api-access-spzw5\") pod \"iptables-alerter-lqm86\" (UID: \"f29fd70f-b10e-4154-8652-6b17fbe3cf13\") " pod="openshift-network-operator/iptables-alerter-lqm86"
Oct 30 19:20:07.977660 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b45e511-0890-4a70-96d7-01ba249d65cb-cni-binary-copy\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:07.977660 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-var-lib-cni-bin\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:07.977660 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977567    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-var-lib-kubelet\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:07.977660 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977608    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-device-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:07.977660 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977629    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbd278f9-76f5-41ed-93d1-435804741834-dbus\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:07.977660 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977643    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-cnibin\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:07.977992 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977665    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-hostroot\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:07.977992 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977679    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv6g9\" (UniqueName: \"kubernetes.io/projected/90c7079a-e03a-4d3e-bb41-b61ce36292ec-kube-api-access-pv6g9\") pod \"node-resolver-n9frl\" (UID: \"90c7079a-e03a-4d3e-bb41-b61ce36292ec\") " pod="openshift-dns/node-resolver-n9frl"
Oct 30 19:20:07.977992 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977694    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-etc-selinux\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:07.977992 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977728    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9zdg\" (UniqueName: \"kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg\") pod \"network-check-target-6dnzx\" (UID: \"c1a81d1f-b828-4a0d-a57a-df3d493947d5\") " pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:07.977992 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977761    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8c299f86-7f6e-4666-ab9c-542537e6ff8e-ovnkube-config\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.977992 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977799    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90c7079a-e03a-4d3e-bb41-b61ce36292ec-tmp-dir\") pod \"node-resolver-n9frl\" (UID: \"90c7079a-e03a-4d3e-bb41-b61ce36292ec\") " pod="openshift-dns/node-resolver-n9frl"
Oct 30 19:20:07.977992 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977856    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aacc7bb9-0109-4984-884f-50f8d2144559-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:07.977992 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977892    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-etc-kubernetes\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:07.977992 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977910    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbd278f9-76f5-41ed-93d1-435804741834-kubelet-config\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:07.977992 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977924    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-var-lib-cni-multus\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:07.977992 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977950    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada-serviceca\") pod \"node-ca-qrsz7\" (UID: \"a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada\") " pod="openshift-image-registry/node-ca-qrsz7"
Oct 30 19:20:07.977992 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.977968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-systemd\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:07.978414 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978000    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f29fd70f-b10e-4154-8652-6b17fbe3cf13-host-slash\") pod \"iptables-alerter-lqm86\" (UID: \"f29fd70f-b10e-4154-8652-6b17fbe3cf13\") " pod="openshift-network-operator/iptables-alerter-lqm86"
Oct 30 19:20:07.978414 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978026    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-run-systemd\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.978414 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978053    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-etc-openvswitch\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.978414 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978084    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-log-socket\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.978414 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978115    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-cni-netd\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.978414 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978138    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8c299f86-7f6e-4666-ab9c-542537e6ff8e-ovnkube-script-lib\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.978414 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978160    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-run-ovn\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.978414 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978203    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bf6h\" (UniqueName: \"kubernetes.io/projected/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada-kube-api-access-4bf6h\") pod \"node-ca-qrsz7\" (UID: \"a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada\") " pod="openshift-image-registry/node-ca-qrsz7"
Oct 30 19:20:07.978414 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978235    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-sysctl-conf\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:07.978414 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978266    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwxjt\" (UniqueName: \"kubernetes.io/projected/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-kube-api-access-gwxjt\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:07.978414 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978326    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:07.978414 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-slash\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.978414 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978419    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-cni-bin\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.978778 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978449    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada-host\") pod \"node-ca-qrsz7\" (UID: \"a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada\") " pod="openshift-image-registry/node-ca-qrsz7"
Oct 30 19:20:07.978778 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-tmp\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:07.978778 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978490    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-var-lib-openvswitch\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.978778 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978504    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-run-openvswitch\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.978778 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978522    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kpwc\" (UniqueName: \"kubernetes.io/projected/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-kube-api-access-5kpwc\") pod \"network-metrics-daemon-vcr5t\" (UID: \"029fdc09-aaca-4dc2-90ff-86e05c26bf9e\") " pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:07.978778 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978562    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs\") pod \"network-metrics-daemon-vcr5t\" (UID: \"029fdc09-aaca-4dc2-90ff-86e05c26bf9e\") " pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:07.978778 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978586    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-conf-dir\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:07.978778 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978628    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:07.978778 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-cni-dir\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:07.978778 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978672    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-socket-dir-parent\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:07.978778 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978693    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-systemd-units\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:07.978778 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978725    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aacc7bb9-0109-4984-884f-50f8d2144559-cnibin\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:07.978778 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978752    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aacc7bb9-0109-4984-884f-50f8d2144559-cni-binary-copy\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:07.978778 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978778    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aacc7bb9-0109-4984-884f-50f8d2144559-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:07.979122 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-sysconfig\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:07.979122 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-kubernetes\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:07.979122 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978852    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-host\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:07.979122 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978892    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-sysctl-d\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:07.979122 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.978925    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-var-lib-kubelet\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:07.986549 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.986526    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:20:07.986650 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.986609    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:20:07.986650 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.986648    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:20:07.986809 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.986789    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:20:07.987013 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.986997    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:20:07.987104 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.987067    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:20:07.987104 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.987089    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:20:07.987179 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.987154    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:20:07.987221 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.987195    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:20:07.987393 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.987287    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9smtf\""
Oct 30 19:20:07.987393 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.987325    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:20:07.987393 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.987338    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:20:07.987563 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.987400    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:20:07.987563 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.987424    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:20:07.987563 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.987521    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b7kzg\""
Oct 30 19:20:07.987699 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.987584    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:20:07.987699 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.987626    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bcv2g\""
Oct 30 19:20:07.987699 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.987634    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gx26w\""
Oct 30 19:20:07.987699 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.987640    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bkrzr\""
Oct 30 19:20:07.987699 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.987636    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-w8vq8\""
Oct 30 19:20:07.987699 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:07.987674    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:20:08.079829 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.079802    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8c299f86-7f6e-4666-ab9c-542537e6ff8e-ovnkube-config\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.079972 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.079834    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90c7079a-e03a-4d3e-bb41-b61ce36292ec-tmp-dir\") pod \"node-resolver-n9frl\" (UID: \"90c7079a-e03a-4d3e-bb41-b61ce36292ec\") " pod="openshift-dns/node-resolver-n9frl"
Oct 30 19:20:08.079972 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.079857    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aacc7bb9-0109-4984-884f-50f8d2144559-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:08.079972 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.079880    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-etc-kubernetes\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.079972 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.079899    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbd278f9-76f5-41ed-93d1-435804741834-kubelet-config\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:08.079972 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.079921    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-var-lib-cni-multus\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.079972 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.079964    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-etc-kubernetes\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.080265 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.079996    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbd278f9-76f5-41ed-93d1-435804741834-kubelet-config\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:08.080265 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada-serviceca\") pod \"node-ca-qrsz7\" (UID: \"a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada\") " pod="openshift-image-registry/node-ca-qrsz7"
Oct 30 19:20:08.080265 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080061    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-systemd\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.080265 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080068    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-var-lib-cni-multus\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.080265 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080098    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f29fd70f-b10e-4154-8652-6b17fbe3cf13-host-slash\") pod \"iptables-alerter-lqm86\" (UID: \"f29fd70f-b10e-4154-8652-6b17fbe3cf13\") " pod="openshift-network-operator/iptables-alerter-lqm86"
Oct 30 19:20:08.080265 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080137    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-systemd\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.080265 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080166    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-run-systemd\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.080265 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080209    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-etc-openvswitch\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.080265 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-log-socket\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.080265 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080240    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90c7079a-e03a-4d3e-bb41-b61ce36292ec-tmp-dir\") pod \"node-resolver-n9frl\" (UID: \"90c7079a-e03a-4d3e-bb41-b61ce36292ec\") " pod="openshift-dns/node-resolver-n9frl"
Oct 30 19:20:08.080743 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080257    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-cni-netd\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.080743 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080267    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f29fd70f-b10e-4154-8652-6b17fbe3cf13-host-slash\") pod \"iptables-alerter-lqm86\" (UID: \"f29fd70f-b10e-4154-8652-6b17fbe3cf13\") " pod="openshift-network-operator/iptables-alerter-lqm86"
Oct 30 19:20:08.080743 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080298    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8c299f86-7f6e-4666-ab9c-542537e6ff8e-ovnkube-script-lib\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.080743 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-run-ovn\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.080743 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080330    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-etc-openvswitch\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.080743 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080331    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-run-systemd\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.080743 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bf6h\" (UniqueName: \"kubernetes.io/projected/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada-kube-api-access-4bf6h\") pod \"node-ca-qrsz7\" (UID: \"a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada\") " pod="openshift-image-registry/node-ca-qrsz7"
Oct 30 19:20:08.080743 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080399    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-log-socket\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.080743 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080412    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-sysctl-conf\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.080743 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080403    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-run-ovn\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.080743 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080437    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwxjt\" (UniqueName: \"kubernetes.io/projected/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-kube-api-access-gwxjt\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.080743 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080453    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-cni-netd\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.080743 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080462    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:08.080743 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080489    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8c299f86-7f6e-4666-ab9c-542537e6ff8e-ovnkube-config\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.080743 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080521    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:08.080743 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080526    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aacc7bb9-0109-4984-884f-50f8d2144559-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:08.080743 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-slash\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.081488 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080567    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada-serviceca\") pod \"node-ca-qrsz7\" (UID: \"a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada\") " pod="openshift-image-registry/node-ca-qrsz7"
Oct 30 19:20:08.081488 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080587    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-cni-bin\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.081488 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080592    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-sysctl-conf\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.081488 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080607    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada-host\") pod \"node-ca-qrsz7\" (UID: \"a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada\") " pod="openshift-image-registry/node-ca-qrsz7"
Oct 30 19:20:08.081488 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080624    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-tmp\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.081488 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080627    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-slash\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.081488 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-var-lib-openvswitch\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.081488 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080646    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-cni-bin\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.081488 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080678    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-var-lib-openvswitch\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.081488 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080677    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada-host\") pod \"node-ca-qrsz7\" (UID: \"a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada\") " pod="openshift-image-registry/node-ca-qrsz7"
Oct 30 19:20:08.081488 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-run-openvswitch\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.081488 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080724    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kpwc\" (UniqueName: \"kubernetes.io/projected/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-kube-api-access-5kpwc\") pod \"network-metrics-daemon-vcr5t\" (UID: \"029fdc09-aaca-4dc2-90ff-86e05c26bf9e\") " pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:08.081488 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080748    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs\") pod \"network-metrics-daemon-vcr5t\" (UID: \"029fdc09-aaca-4dc2-90ff-86e05c26bf9e\") " pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:08.081488 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080772    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-conf-dir\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.081488 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080765    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-run-openvswitch\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.081488 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080800    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:08.081488 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-cni-dir\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.081488 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080845    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-conf-dir\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.082307 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:08.080862    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:20:08.082307 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080923    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8c299f86-7f6e-4666-ab9c-542537e6ff8e-ovnkube-script-lib\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.082307 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080850    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-socket-dir-parent\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.082307 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:08.080929    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:20:08.082307 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:08.080955    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs podName:029fdc09-aaca-4dc2-90ff-86e05c26bf9e nodeName:}" failed. No retries permitted until 2026-10-30 19:20:08.580905839 +0000 UTC m=+3.065561952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs") pod "network-metrics-daemon-vcr5t" (UID: "029fdc09-aaca-4dc2-90ff-86e05c26bf9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:20:08.082307 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080975    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-socket-dir-parent\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.082307 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080994    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-cni-dir\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.082307 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:08.081011    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret podName:bbd278f9-76f5-41ed-93d1-435804741834 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:08.580998206 +0000 UTC m=+3.065654313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret") pod "global-pull-secret-syncer-27xsq" (UID: "bbd278f9-76f5-41ed-93d1-435804741834") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:20:08.082307 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081032    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-systemd-units\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.082307 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aacc7bb9-0109-4984-884f-50f8d2144559-cnibin\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:08.082307 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.080937    2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:20:08.082307 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081085    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aacc7bb9-0109-4984-884f-50f8d2144559-cni-binary-copy\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:08.082307 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081113    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aacc7bb9-0109-4984-884f-50f8d2144559-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:08.082307 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081143    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-sysconfig\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.082307 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081147    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aacc7bb9-0109-4984-884f-50f8d2144559-cnibin\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:08.082307 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081180    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-kubernetes\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.082307 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081205    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-host\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.082937 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081221    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-sysconfig\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.082937 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081227    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-sysctl-d\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.082937 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081250    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-var-lib-kubelet\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.082937 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081267    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/954bf0fe-4287-4435-9abb-fd06dc478099-agent-certs\") pod \"konnectivity-agent-l65sf\" (UID: \"954bf0fe-4287-4435-9abb-fd06dc478099\") " pod="kube-system/konnectivity-agent-l65sf"
Oct 30 19:20:08.082937 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081284    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-system-cni-dir\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.082937 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081299    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-daemon-config\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.082937 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081314    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-run-netns\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.082937 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081327    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-sysctl-d\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.082937 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081114    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-systemd-units\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.082937 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081390    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-node-log\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.082937 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081395    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-var-lib-kubelet\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.082937 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081436    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-kubernetes\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.082937 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081332    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-node-log\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.082937 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081491    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-system-cni-dir\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.082937 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081498    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-socket-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:08.082937 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081513    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-run-netns\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.082937 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081264    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aacc7bb9-0109-4984-884f-50f8d2144559-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:08.082937 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aacc7bb9-0109-4984-884f-50f8d2144559-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:08.083735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081582    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-lib-modules\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.083735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081592    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-socket-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:08.083735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081608    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqxbc\" (UniqueName: \"kubernetes.io/projected/0b45e511-0890-4a70-96d7-01ba249d65cb-kube-api-access-lqxbc\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.083735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081638    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-kubelet\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.083735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081658    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-host\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.083735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-registration-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:08.083735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-sys-fs\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:08.083735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081707    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-registration-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:08.083735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081711    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-lib-modules\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.083735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081724    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkgsr\" (UniqueName: \"kubernetes.io/projected/093cd549-ebec-45f5-b888-42019476c70c-kube-api-access-tkgsr\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:08.083735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081728    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aacc7bb9-0109-4984-884f-50f8d2144559-cni-binary-copy\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:08.083735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081779    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aacc7bb9-0109-4984-884f-50f8d2144559-system-cni-dir\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:08.083735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081868    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-sys-fs\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:08.083735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081912    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-modprobe-d\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.083735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081942    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aacc7bb9-0109-4984-884f-50f8d2144559-system-cni-dir\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:08.083735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081955    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-kubelet\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.083735 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.081978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-run\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.084290 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082012    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f29fd70f-b10e-4154-8652-6b17fbe3cf13-iptables-alerter-script\") pod \"iptables-alerter-lqm86\" (UID: \"f29fd70f-b10e-4154-8652-6b17fbe3cf13\") " pod="openshift-network-operator/iptables-alerter-lqm86"
Oct 30 19:20:08.084290 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082038    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-modprobe-d\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.084290 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082049    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-run\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.084290 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082081    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aacc7bb9-0109-4984-884f-50f8d2144559-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:08.084290 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082087    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-daemon-config\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.084290 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082123    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-run-netns\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.084290 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-run-netns\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.084290 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-run-ovn-kubernetes\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.084290 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082256    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8c299f86-7f6e-4666-ab9c-542537e6ff8e-env-overrides\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.084290 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082276    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-run-ovn-kubernetes\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.084290 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90c7079a-e03a-4d3e-bb41-b61ce36292ec-hosts-file\") pod \"node-resolver-n9frl\" (UID: \"90c7079a-e03a-4d3e-bb41-b61ce36292ec\") " pod="openshift-dns/node-resolver-n9frl"
Oct 30 19:20:08.084290 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082356    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-sys\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.084290 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082401    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90c7079a-e03a-4d3e-bb41-b61ce36292ec-hosts-file\") pod \"node-resolver-n9frl\" (UID: \"90c7079a-e03a-4d3e-bb41-b61ce36292ec\") " pod="openshift-dns/node-resolver-n9frl"
Oct 30 19:20:08.084290 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082490    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-run-multus-certs\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.084290 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082553    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-sys\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.084290 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082635    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f29fd70f-b10e-4154-8652-6b17fbe3cf13-iptables-alerter-script\") pod \"iptables-alerter-lqm86\" (UID: \"f29fd70f-b10e-4154-8652-6b17fbe3cf13\") " pod="openshift-network-operator/iptables-alerter-lqm86"
Oct 30 19:20:08.084290 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082744    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8c299f86-7f6e-4666-ab9c-542537e6ff8e-env-overrides\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.084290 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-run-multus-certs\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.085007 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082907    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/954bf0fe-4287-4435-9abb-fd06dc478099-konnectivity-ca\") pod \"konnectivity-agent-l65sf\" (UID: \"954bf0fe-4287-4435-9abb-fd06dc478099\") " pod="kube-system/konnectivity-agent-l65sf"
Oct 30 19:20:08.085007 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082933    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-run-k8s-cni-cncf-io\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.085007 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082957    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drlzm\" (UniqueName: \"kubernetes.io/projected/aacc7bb9-0109-4984-884f-50f8d2144559-kube-api-access-drlzm\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:08.085007 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.082981    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-tuned\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.085007 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.083014    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.085007 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.083041    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c299f86-7f6e-4666-ab9c-542537e6ff8e-ovn-node-metrics-cert\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.085007 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.083064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5xl5\" (UniqueName: \"kubernetes.io/projected/8c299f86-7f6e-4666-ab9c-542537e6ff8e-kube-api-access-d5xl5\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.085007 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.083088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spzw5\" (UniqueName: \"kubernetes.io/projected/f29fd70f-b10e-4154-8652-6b17fbe3cf13-kube-api-access-spzw5\") pod \"iptables-alerter-lqm86\" (UID: \"f29fd70f-b10e-4154-8652-6b17fbe3cf13\") " pod="openshift-network-operator/iptables-alerter-lqm86"
Oct 30 19:20:08.085007 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.083113    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b45e511-0890-4a70-96d7-01ba249d65cb-cni-binary-copy\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.085007 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.083135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-var-lib-cni-bin\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.085007 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.083159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-var-lib-kubelet\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.085007 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.083182    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-device-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:08.085007 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.083206    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbd278f9-76f5-41ed-93d1-435804741834-dbus\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:08.085007 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.083230    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-cnibin\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.085007 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.083253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-hostroot\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.085007 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.083277    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv6g9\" (UniqueName: \"kubernetes.io/projected/90c7079a-e03a-4d3e-bb41-b61ce36292ec-kube-api-access-pv6g9\") pod \"node-resolver-n9frl\" (UID: \"90c7079a-e03a-4d3e-bb41-b61ce36292ec\") " pod="openshift-dns/node-resolver-n9frl"
Oct 30 19:20:08.085007 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.083303    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-etc-selinux\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:08.085618 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.083327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9zdg\" (UniqueName: \"kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg\") pod \"network-check-target-6dnzx\" (UID: \"c1a81d1f-b828-4a0d-a57a-df3d493947d5\") " pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:08.085618 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.084039    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/954bf0fe-4287-4435-9abb-fd06dc478099-konnectivity-ca\") pod \"konnectivity-agent-l65sf\" (UID: \"954bf0fe-4287-4435-9abb-fd06dc478099\") " pod="kube-system/konnectivity-agent-l65sf"
Oct 30 19:20:08.085618 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.084133    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-run-k8s-cni-cncf-io\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.085618 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.084584    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-var-lib-kubelet\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.085618 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.084640    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.085618 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.085270    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/954bf0fe-4287-4435-9abb-fd06dc478099-agent-certs\") pod \"konnectivity-agent-l65sf\" (UID: \"954bf0fe-4287-4435-9abb-fd06dc478099\") " pod="kube-system/konnectivity-agent-l65sf"
Oct 30 19:20:08.085618 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.085395    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-cnibin\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.085618 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.085448    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-device-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:08.085618 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.085505    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbd278f9-76f5-41ed-93d1-435804741834-dbus\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:08.086145 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.086125    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-hostroot\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.086221 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.086194    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-var-lib-cni-bin\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.086295 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.086278    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-etc-selinux\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:08.086495 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.086471    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-tmp\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.086582 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.086498    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b45e511-0890-4a70-96d7-01ba249d65cb-cni-binary-copy\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.086959 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.086940    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c299f86-7f6e-4666-ab9c-542537e6ff8e-ovn-node-metrics-cert\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.087593 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.087576    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-tuned\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.100856 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.100752    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwxjt\" (UniqueName: \"kubernetes.io/projected/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-kube-api-access-gwxjt\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.101326 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:08.100940    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:20:08.101326 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:08.100994    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:20:08.101326 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:08.101007    2582 projected.go:196] Error preparing data for projected volume kube-api-access-r9zdg for pod openshift-network-diagnostics/network-check-target-6dnzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:20:08.101326 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:08.101059    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg podName:c1a81d1f-b828-4a0d-a57a-df3d493947d5 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:08.601043197 +0000 UTC m=+3.085699301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r9zdg" (UniqueName: "kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg") pod "network-check-target-6dnzx" (UID: "c1a81d1f-b828-4a0d-a57a-df3d493947d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:20:08.102315 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.102287    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bf6h\" (UniqueName: \"kubernetes.io/projected/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada-kube-api-access-4bf6h\") pod \"node-ca-qrsz7\" (UID: \"a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada\") " pod="openshift-image-registry/node-ca-qrsz7"
Oct 30 19:20:08.103053 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.103028    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqxbc\" (UniqueName: \"kubernetes.io/projected/0b45e511-0890-4a70-96d7-01ba249d65cb-kube-api-access-lqxbc\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.103117 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.103094    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkgsr\" (UniqueName: \"kubernetes.io/projected/093cd549-ebec-45f5-b888-42019476c70c-kube-api-access-tkgsr\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:08.105245 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.105193    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drlzm\" (UniqueName: \"kubernetes.io/projected/aacc7bb9-0109-4984-884f-50f8d2144559-kube-api-access-drlzm\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:08.106031 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.105536    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kpwc\" (UniqueName: \"kubernetes.io/projected/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-kube-api-access-5kpwc\") pod \"network-metrics-daemon-vcr5t\" (UID: \"029fdc09-aaca-4dc2-90ff-86e05c26bf9e\") " pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:08.106689 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.106420    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal" event={"ID":"2670e93fcc64eb8a9f070c9bd973c8ad","Type":"ContainerStarted","Data":"0008bcf05fe731a028bb8218b6d25cc4481c6f5e08353eed82e64ec4cf016693"}
Oct 30 19:20:08.107200 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.107173    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spzw5\" (UniqueName: \"kubernetes.io/projected/f29fd70f-b10e-4154-8652-6b17fbe3cf13-kube-api-access-spzw5\") pod \"iptables-alerter-lqm86\" (UID: \"f29fd70f-b10e-4154-8652-6b17fbe3cf13\") " pod="openshift-network-operator/iptables-alerter-lqm86"
Oct 30 19:20:08.107285 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.107197    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5xl5\" (UniqueName: \"kubernetes.io/projected/8c299f86-7f6e-4666-ab9c-542537e6ff8e-kube-api-access-d5xl5\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.108828 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.108673    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv6g9\" (UniqueName: \"kubernetes.io/projected/90c7079a-e03a-4d3e-bb41-b61ce36292ec-kube-api-access-pv6g9\") pod \"node-resolver-n9frl\" (UID: \"90c7079a-e03a-4d3e-bb41-b61ce36292ec\") " pod="openshift-dns/node-resolver-n9frl"
Oct 30 19:20:08.109435 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.109267    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal" event={"ID":"4e9154b2294c90c44348210589c7863a","Type":"ContainerStarted","Data":"8bc6f2de22212ef491069993807ee1b09446c52f5dee4c880b8e8f28a5c4cb40"}
Oct 30 19:20:08.180472 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.180444    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:20:08.189935 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.189907    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:08.200554 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:08.200525    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b45e511_0890_4a70_96d7_01ba249d65cb.slice/crio-8f71ec68a3bf3f19f744fb775f8ec2b1f2b8ac746ea745d970c96f5057a3c01d WatchSource:0}: Error finding container 8f71ec68a3bf3f19f744fb775f8ec2b1f2b8ac746ea745d970c96f5057a3c01d: Status 404 returned error can't find the container with id 8f71ec68a3bf3f19f744fb775f8ec2b1f2b8ac746ea745d970c96f5057a3c01d
Oct 30 19:20:08.210177 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.210151    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:08.214388 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.214357    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n9frl"
Oct 30 19:20:08.217646 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.217626    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:08.218995 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:08.218971    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4aa3418_bbc0_4c70_a42a_2a65e478b47d.slice/crio-7e1d5eeb37d21639a88877f838aea15d395de840268d98c45ef047ed66dd99db WatchSource:0}: Error finding container 7e1d5eeb37d21639a88877f838aea15d395de840268d98c45ef047ed66dd99db: Status 404 returned error can't find the container with id 7e1d5eeb37d21639a88877f838aea15d395de840268d98c45ef047ed66dd99db
Oct 30 19:20:08.219609 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.219588    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qrsz7"
Oct 30 19:20:08.227860 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:08.227837    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c7079a_e03a_4d3e_bb41_b61ce36292ec.slice/crio-44a177bde116a168d2eef7e8125c71aa8e9cff40c9c22201148e20d92b106799 WatchSource:0}: Error finding container 44a177bde116a168d2eef7e8125c71aa8e9cff40c9c22201148e20d92b106799: Status 404 returned error can't find the container with id 44a177bde116a168d2eef7e8125c71aa8e9cff40c9c22201148e20d92b106799
Oct 30 19:20:08.228596 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.228573    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:08.229874 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:08.229841    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c299f86_7f6e_4666_ab9c_542537e6ff8e.slice/crio-d60f6c831ff3e931fe18c1fc7736ba7c15f83287f202a5f954397eaa96a3c8b9 WatchSource:0}: Error finding container d60f6c831ff3e931fe18c1fc7736ba7c15f83287f202a5f954397eaa96a3c8b9: Status 404 returned error can't find the container with id d60f6c831ff3e931fe18c1fc7736ba7c15f83287f202a5f954397eaa96a3c8b9
Oct 30 19:20:08.234417 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:08.234391    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e454ca_29fa_4b1d_8c8d_5db1c6c6cada.slice/crio-c727fc80a4e6ec4e17ef4cd84e37e14f8d6ba36d4230b5d05e4932ae8e33b3dc WatchSource:0}: Error finding container c727fc80a4e6ec4e17ef4cd84e37e14f8d6ba36d4230b5d05e4932ae8e33b3dc: Status 404 returned error can't find the container with id c727fc80a4e6ec4e17ef4cd84e37e14f8d6ba36d4230b5d05e4932ae8e33b3dc
Oct 30 19:20:08.244104 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.244083    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lqm86"
Oct 30 19:20:08.245204 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.245184    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l65sf"
Oct 30 19:20:08.247520 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.247460    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:08.254023 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:08.254002    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29fd70f_b10e_4154_8652_6b17fbe3cf13.slice/crio-87f8f4d5303e13435b5f3db595fc92e5b67ca5791dab7b732ed6ecb60cd55374 WatchSource:0}: Error finding container 87f8f4d5303e13435b5f3db595fc92e5b67ca5791dab7b732ed6ecb60cd55374: Status 404 returned error can't find the container with id 87f8f4d5303e13435b5f3db595fc92e5b67ca5791dab7b732ed6ecb60cd55374
Oct 30 19:20:08.259552 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:08.259527    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacc7bb9_0109_4984_884f_50f8d2144559.slice/crio-25a1ca4031e90d820f747872f6f21e62c65560f9ac1f5bec98d59483f52873d9 WatchSource:0}: Error finding container 25a1ca4031e90d820f747872f6f21e62c65560f9ac1f5bec98d59483f52873d9: Status 404 returned error can't find the container with id 25a1ca4031e90d820f747872f6f21e62c65560f9ac1f5bec98d59483f52873d9
Oct 30 19:20:08.261936 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:08.261911    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954bf0fe_4287_4435_9abb_fd06dc478099.slice/crio-0889efca5dbac94607f60d7ed64589805be039dcfd58f4d2d47c84eefa116e07 WatchSource:0}: Error finding container 0889efca5dbac94607f60d7ed64589805be039dcfd58f4d2d47c84eefa116e07: Status 404 returned error can't find the container with id 0889efca5dbac94607f60d7ed64589805be039dcfd58f4d2d47c84eefa116e07
Oct 30 19:20:08.407036 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.407009    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:20:08.586634 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.586562    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs\") pod \"network-metrics-daemon-vcr5t\" (UID: \"029fdc09-aaca-4dc2-90ff-86e05c26bf9e\") " pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:08.586634 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.586609    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:08.586850 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:08.586738    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:20:08.586850 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:08.586794    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:20:08.586923 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:08.586815    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs podName:029fdc09-aaca-4dc2-90ff-86e05c26bf9e nodeName:}" failed. No retries permitted until 2026-10-30 19:20:09.586791511 +0000 UTC m=+4.071447612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs") pod "network-metrics-daemon-vcr5t" (UID: "029fdc09-aaca-4dc2-90ff-86e05c26bf9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:20:08.586923 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:08.586885    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret podName:bbd278f9-76f5-41ed-93d1-435804741834 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:09.586866585 +0000 UTC m=+4.071522683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret") pod "global-pull-secret-syncer-27xsq" (UID: "bbd278f9-76f5-41ed-93d1-435804741834") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:20:08.690172 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.689642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9zdg\" (UniqueName: \"kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg\") pod \"network-check-target-6dnzx\" (UID: \"c1a81d1f-b828-4a0d-a57a-df3d493947d5\") " pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:08.690172 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:08.689776    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:20:08.690172 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:08.689800    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:20:08.690172 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:08.689811    2582 projected.go:196] Error preparing data for projected volume kube-api-access-r9zdg for pod openshift-network-diagnostics/network-check-target-6dnzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:20:08.690172 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:08.689869    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg podName:c1a81d1f-b828-4a0d-a57a-df3d493947d5 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:09.689852163 +0000 UTC m=+4.174508266 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r9zdg" (UniqueName: "kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg") pod "network-check-target-6dnzx" (UID: "c1a81d1f-b828-4a0d-a57a-df3d493947d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:20:08.913998 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.913911    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:15:07 +0000 UTC" deadline="2028-05-08 02:52:18.783152557 +0000 UTC"
Oct 30 19:20:08.913998 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:08.913947    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13327h32m9.869209541s"
Oct 30 19:20:09.099542 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:09.099509    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:09.099716 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:09.099637    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27xsq" podUID="bbd278f9-76f5-41ed-93d1-435804741834"
Oct 30 19:20:09.117004 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:09.116970    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bw8v2" event={"ID":"0b45e511-0890-4a70-96d7-01ba249d65cb","Type":"ContainerStarted","Data":"8f71ec68a3bf3f19f744fb775f8ec2b1f2b8ac746ea745d970c96f5057a3c01d"}
Oct 30 19:20:09.133895 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:09.133860    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jjm2b" event={"ID":"aacc7bb9-0109-4984-884f-50f8d2144559","Type":"ContainerStarted","Data":"25a1ca4031e90d820f747872f6f21e62c65560f9ac1f5bec98d59483f52873d9"}
Oct 30 19:20:09.138498 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:09.138469    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lqm86" event={"ID":"f29fd70f-b10e-4154-8652-6b17fbe3cf13","Type":"ContainerStarted","Data":"87f8f4d5303e13435b5f3db595fc92e5b67ca5791dab7b732ed6ecb60cd55374"}
Oct 30 19:20:09.140618 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:09.140537    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n" event={"ID":"093cd549-ebec-45f5-b888-42019476c70c","Type":"ContainerStarted","Data":"6a2d80d43f8e032c61835cfefd5452762660b8d3b2b040861342d06f421cdf04"}
Oct 30 19:20:09.150585 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:09.149873    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal" event={"ID":"4e9154b2294c90c44348210589c7863a","Type":"ContainerStarted","Data":"3b706d55464f0ae0908571c9c2c821e2370728ba1e55ac42e5ff2ef1be76d1d2"}
Oct 30 19:20:09.153136 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:09.153110    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l65sf" event={"ID":"954bf0fe-4287-4435-9abb-fd06dc478099","Type":"ContainerStarted","Data":"0889efca5dbac94607f60d7ed64589805be039dcfd58f4d2d47c84eefa116e07"}
Oct 30 19:20:09.169805 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:09.169660    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qrsz7" event={"ID":"a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada","Type":"ContainerStarted","Data":"c727fc80a4e6ec4e17ef4cd84e37e14f8d6ba36d4230b5d05e4932ae8e33b3dc"}
Oct 30 19:20:09.173585 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:09.173553    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc" event={"ID":"8c299f86-7f6e-4666-ab9c-542537e6ff8e","Type":"ContainerStarted","Data":"d60f6c831ff3e931fe18c1fc7736ba7c15f83287f202a5f954397eaa96a3c8b9"}
Oct 30 19:20:09.177196 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:09.177146    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n9frl" event={"ID":"90c7079a-e03a-4d3e-bb41-b61ce36292ec","Type":"ContainerStarted","Data":"44a177bde116a168d2eef7e8125c71aa8e9cff40c9c22201148e20d92b106799"}
Oct 30 19:20:09.187785 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:09.187760    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2gf9g" event={"ID":"c4aa3418-bbc0-4c70-a42a-2a65e478b47d","Type":"ContainerStarted","Data":"7e1d5eeb37d21639a88877f838aea15d395de840268d98c45ef047ed66dd99db"}
Oct 30 19:20:09.190700 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:09.190647    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal" podStartSLOduration=2.190631801 podStartE2EDuration="2.190631801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:09.189620829 +0000 UTC m=+3.674276934" watchObservedRunningTime="2026-10-30 19:20:09.190631801 +0000 UTC m=+3.675287913"
Oct 30 19:20:09.596072 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:09.595966    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs\") pod \"network-metrics-daemon-vcr5t\" (UID: \"029fdc09-aaca-4dc2-90ff-86e05c26bf9e\") " pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:09.596072 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:09.596008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:09.596347 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:09.596165    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:20:09.596347 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:09.596221    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret podName:bbd278f9-76f5-41ed-93d1-435804741834 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:11.596205304 +0000 UTC m=+6.080861396 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret") pod "global-pull-secret-syncer-27xsq" (UID: "bbd278f9-76f5-41ed-93d1-435804741834") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:20:09.596612 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:09.596594    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:20:09.596686 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:09.596643    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs podName:029fdc09-aaca-4dc2-90ff-86e05c26bf9e nodeName:}" failed. No retries permitted until 2026-10-30 19:20:11.596628556 +0000 UTC m=+6.081284663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs") pod "network-metrics-daemon-vcr5t" (UID: "029fdc09-aaca-4dc2-90ff-86e05c26bf9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:20:09.697302 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:09.697270    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9zdg\" (UniqueName: \"kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg\") pod \"network-check-target-6dnzx\" (UID: \"c1a81d1f-b828-4a0d-a57a-df3d493947d5\") " pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:09.697480 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:09.697463    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:20:09.697547 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:09.697491    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:20:09.697547 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:09.697504    2582 projected.go:196] Error preparing data for projected volume kube-api-access-r9zdg for pod openshift-network-diagnostics/network-check-target-6dnzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:20:09.697643 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:09.697565    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg podName:c1a81d1f-b828-4a0d-a57a-df3d493947d5 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:11.697546722 +0000 UTC m=+6.182202832 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r9zdg" (UniqueName: "kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg") pod "network-check-target-6dnzx" (UID: "c1a81d1f-b828-4a0d-a57a-df3d493947d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:20:09.914608 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:09.914336    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:15:07 +0000 UTC" deadline="2028-03-28 08:15:49.68705615 +0000 UTC"
Oct 30 19:20:09.914608 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:09.914401    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12348h55m39.772659343s"
Oct 30 19:20:10.101961 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:10.101928    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:10.102128 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:10.102060    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcr5t" podUID="029fdc09-aaca-4dc2-90ff-86e05c26bf9e"
Oct 30 19:20:10.102474 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:10.102452    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:10.102569 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:10.102550    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnzx" podUID="c1a81d1f-b828-4a0d-a57a-df3d493947d5"
Oct 30 19:20:11.099169 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:11.099144    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:11.099615 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:11.099259    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27xsq" podUID="bbd278f9-76f5-41ed-93d1-435804741834"
Oct 30 19:20:11.204388 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:11.203575    2582 generic.go:356] "Generic (PLEG): container finished" podID="2670e93fcc64eb8a9f070c9bd973c8ad" containerID="11a15e06b0d4eb386807e9a1fdf660153c2f718eca5719db9b858af0e675796c" exitCode=0
Oct 30 19:20:11.204388 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:11.203660    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal" event={"ID":"2670e93fcc64eb8a9f070c9bd973c8ad","Type":"ContainerDied","Data":"11a15e06b0d4eb386807e9a1fdf660153c2f718eca5719db9b858af0e675796c"}
Oct 30 19:20:11.213436 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:11.212881    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jjm2b" event={"ID":"aacc7bb9-0109-4984-884f-50f8d2144559","Type":"ContainerStarted","Data":"1560f92745fe53548214df6cb0904495acd37d96e0a81b5b4323c6773433b8e3"}
Oct 30 19:20:11.615886 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:11.615163    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs\") pod \"network-metrics-daemon-vcr5t\" (UID: \"029fdc09-aaca-4dc2-90ff-86e05c26bf9e\") " pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:11.615886 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:11.615208    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:11.615886 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:11.615382    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:20:11.615886 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:11.615443    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret podName:bbd278f9-76f5-41ed-93d1-435804741834 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:15.615424967 +0000 UTC m=+10.100081061 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret") pod "global-pull-secret-syncer-27xsq" (UID: "bbd278f9-76f5-41ed-93d1-435804741834") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:20:11.615886 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:11.615806    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:20:11.615886 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:11.615853    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs podName:029fdc09-aaca-4dc2-90ff-86e05c26bf9e nodeName:}" failed. No retries permitted until 2026-10-30 19:20:15.61583858 +0000 UTC m=+10.100494683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs") pod "network-metrics-daemon-vcr5t" (UID: "029fdc09-aaca-4dc2-90ff-86e05c26bf9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:20:11.717017 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:11.716384    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9zdg\" (UniqueName: \"kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg\") pod \"network-check-target-6dnzx\" (UID: \"c1a81d1f-b828-4a0d-a57a-df3d493947d5\") " pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:11.717017 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:11.716553    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:20:11.717017 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:11.716570    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:20:11.717017 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:11.716583    2582 projected.go:196] Error preparing data for projected volume kube-api-access-r9zdg for pod openshift-network-diagnostics/network-check-target-6dnzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:20:11.717017 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:11.716638    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg podName:c1a81d1f-b828-4a0d-a57a-df3d493947d5 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:15.716621354 +0000 UTC m=+10.201277457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r9zdg" (UniqueName: "kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg") pod "network-check-target-6dnzx" (UID: "c1a81d1f-b828-4a0d-a57a-df3d493947d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:20:12.099321 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:12.099249    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:12.099774 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:12.099399    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcr5t" podUID="029fdc09-aaca-4dc2-90ff-86e05c26bf9e"
Oct 30 19:20:12.099852 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:12.099772    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:12.099900 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:12.099863    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnzx" podUID="c1a81d1f-b828-4a0d-a57a-df3d493947d5"
Oct 30 19:20:12.231114 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:12.231077    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal" event={"ID":"2670e93fcc64eb8a9f070c9bd973c8ad","Type":"ContainerStarted","Data":"466f6f568ef20c0f46bde94294aea51f826c1fff4167a1e9c1aac92e63ba5bc5"}
Oct 30 19:20:12.237872 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:12.237841    2582 generic.go:356] "Generic (PLEG): container finished" podID="aacc7bb9-0109-4984-884f-50f8d2144559" containerID="1560f92745fe53548214df6cb0904495acd37d96e0a81b5b4323c6773433b8e3" exitCode=0
Oct 30 19:20:12.237985 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:12.237891    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jjm2b" event={"ID":"aacc7bb9-0109-4984-884f-50f8d2144559","Type":"ContainerDied","Data":"1560f92745fe53548214df6cb0904495acd37d96e0a81b5b4323c6773433b8e3"}
Oct 30 19:20:12.278415 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:12.278334    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal" podStartSLOduration=5.278317082 podStartE2EDuration="5.278317082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:12.271052035 +0000 UTC m=+6.755708147" watchObservedRunningTime="2026-10-30 19:20:12.278317082 +0000 UTC m=+6.762973196"
Oct 30 19:20:13.099378 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:13.099334    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:13.099835 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:13.099531    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27xsq" podUID="bbd278f9-76f5-41ed-93d1-435804741834"
Oct 30 19:20:13.243729 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:13.243651    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l65sf" event={"ID":"954bf0fe-4287-4435-9abb-fd06dc478099","Type":"ContainerStarted","Data":"9834f20fc2e7e6834fb07ea460b86b0830b5f8c466c4a007f4b5a783dd232e38"}
Oct 30 19:20:14.104105 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:14.104069    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:14.104654 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:14.104193    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnzx" podUID="c1a81d1f-b828-4a0d-a57a-df3d493947d5"
Oct 30 19:20:14.104654 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:14.104581    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:14.104784 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:14.104687    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcr5t" podUID="029fdc09-aaca-4dc2-90ff-86e05c26bf9e"
Oct 30 19:20:14.250046 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:14.249320    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qrsz7" event={"ID":"a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada","Type":"ContainerStarted","Data":"e1c0770c5434b5c28ce0647a7f66462b79213e1dc5cc1d185a27ca0693d42b45"}
Oct 30 19:20:14.252878 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:14.252805    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n" event={"ID":"093cd549-ebec-45f5-b888-42019476c70c","Type":"ContainerStarted","Data":"40617d89efc54248baf4d25ff80d73ba180d71f524cb8d01c38aff4a831bb0fa"}
Oct 30 19:20:14.287145 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:14.286744    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l65sf" podStartSLOduration=4.375995076 podStartE2EDuration="8.286725955s" totalImagesPullingTime="3.910730879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:13.282392409 +0000 UTC m=+7.767048520" watchObservedRunningTime="2026-10-30 19:20:14.286725955 +0000 UTC m=+8.771382067"
Oct 30 19:20:15.099281 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:15.099199    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:15.099457 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:15.099327    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27xsq" podUID="bbd278f9-76f5-41ed-93d1-435804741834"
Oct 30 19:20:15.647228 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:15.647148    2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:20:15.651096 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:15.651070    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs\") pod \"network-metrics-daemon-vcr5t\" (UID: \"029fdc09-aaca-4dc2-90ff-86e05c26bf9e\") " pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:15.651170 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:15.651114    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:15.651293 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:15.651270    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:20:15.651380 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:15.651332    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret podName:bbd278f9-76f5-41ed-93d1-435804741834 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:23.651305628 +0000 UTC m=+18.135961731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret") pod "global-pull-secret-syncer-27xsq" (UID: "bbd278f9-76f5-41ed-93d1-435804741834") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:20:15.651439 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:15.651415    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:20:15.651558 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:15.651524    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs podName:029fdc09-aaca-4dc2-90ff-86e05c26bf9e nodeName:}" failed. No retries permitted until 2026-10-30 19:20:23.651509695 +0000 UTC m=+18.136165798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs") pod "network-metrics-daemon-vcr5t" (UID: "029fdc09-aaca-4dc2-90ff-86e05c26bf9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:20:15.751794 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:15.751711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9zdg\" (UniqueName: \"kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg\") pod \"network-check-target-6dnzx\" (UID: \"c1a81d1f-b828-4a0d-a57a-df3d493947d5\") " pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:15.751997 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:15.751863    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:20:15.751997 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:15.751884    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:20:15.751997 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:15.751896    2582 projected.go:196] Error preparing data for projected volume kube-api-access-r9zdg for pod openshift-network-diagnostics/network-check-target-6dnzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:20:15.751997 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:15.751947    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg podName:c1a81d1f-b828-4a0d-a57a-df3d493947d5 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:23.75193115 +0000 UTC m=+18.236587254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r9zdg" (UniqueName: "kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg") pod "network-check-target-6dnzx" (UID: "c1a81d1f-b828-4a0d-a57a-df3d493947d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:20:16.060830 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:16.060727    2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:20:15.647168444Z","UUID":"5a0cec7a-81f1-4c47-bdd2-c4e74c6314ad","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:20:16.066176 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:16.063466    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:20:16.066176 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:16.063527    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:20:16.101586 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:16.101514    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:16.101751 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:16.101727    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnzx" podUID="c1a81d1f-b828-4a0d-a57a-df3d493947d5"
Oct 30 19:20:16.102408 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:16.102390    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:16.102751 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:16.102691    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcr5t" podUID="029fdc09-aaca-4dc2-90ff-86e05c26bf9e"
Oct 30 19:20:16.261047 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:16.260312    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n9frl" event={"ID":"90c7079a-e03a-4d3e-bb41-b61ce36292ec","Type":"ContainerStarted","Data":"ad96625b97722951b3229f49726057469e095a50c33c171e6bbc35d77240786a"}
Oct 30 19:20:16.263498 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:16.263092    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bw8v2" event={"ID":"0b45e511-0890-4a70-96d7-01ba249d65cb","Type":"ContainerStarted","Data":"ba59b1a2dbf7764bce6f38b4955afc7d57a8286e7ed93c5f4cf9afe00a907aa0"}
Oct 30 19:20:16.266432 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:16.266213    2582 generic.go:356] "Generic (PLEG): container finished" podID="aacc7bb9-0109-4984-884f-50f8d2144559" containerID="348a4b52a691d67d14c50f9ceccb82a7bc847707dae1610a7a48a6090541290c" exitCode=0
Oct 30 19:20:16.266432 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:16.266272    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jjm2b" event={"ID":"aacc7bb9-0109-4984-884f-50f8d2144559","Type":"ContainerDied","Data":"348a4b52a691d67d14c50f9ceccb82a7bc847707dae1610a7a48a6090541290c"}
Oct 30 19:20:16.269725 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:16.269658    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n" event={"ID":"093cd549-ebec-45f5-b888-42019476c70c","Type":"ContainerStarted","Data":"074f00f23deb4ccb878c5bca2619240f096b54dabbdb38cc99e2598ae20eaf87"}
Oct 30 19:20:16.298102 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:16.298018    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qrsz7" podStartSLOduration=5.249791603 podStartE2EDuration="10.298003395s" totalImagesPullingTime="5.048211792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:14.287104262 +0000 UTC m=+8.771760375" watchObservedRunningTime="2026-10-30 19:20:16.298003395 +0000 UTC m=+10.782659507"
Oct 30 19:20:16.341738 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:16.341682    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-n9frl" podStartSLOduration=2.6016878070000002 podStartE2EDuration="10.341663107s" totalImagesPullingTime="7.7399753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:16.299070154 +0000 UTC m=+10.783726266" watchObservedRunningTime="2026-10-30 19:20:16.341663107 +0000 UTC m=+10.826319219"
Oct 30 19:20:16.383723 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:16.383343    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-bw8v2" podStartSLOduration=2.59843858 podStartE2EDuration="10.3833288s" totalImagesPullingTime="7.78489022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:16.382688316 +0000 UTC m=+10.867344422" watchObservedRunningTime="2026-10-30 19:20:16.3833288 +0000 UTC m=+10.867984912"
Oct 30 19:20:17.100197 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:17.099957    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:17.100782 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:17.100291    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27xsq" podUID="bbd278f9-76f5-41ed-93d1-435804741834"
Oct 30 19:20:17.274678 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:17.274598    2582 generic.go:356] "Generic (PLEG): container finished" podID="aacc7bb9-0109-4984-884f-50f8d2144559" containerID="def8dedaa8407ed4143926c9fe8aa64e334b326e8cce6933728b39c45ebabfd1" exitCode=0
Oct 30 19:20:17.274819 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:17.274689    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jjm2b" event={"ID":"aacc7bb9-0109-4984-884f-50f8d2144559","Type":"ContainerDied","Data":"def8dedaa8407ed4143926c9fe8aa64e334b326e8cce6933728b39c45ebabfd1"}
Oct 30 19:20:17.279630 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:17.279604    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n" event={"ID":"093cd549-ebec-45f5-b888-42019476c70c","Type":"ContainerStarted","Data":"2e4f5c31de1b2080de0bd3e5741a7ad78cb42480239f0ddd9d20185768c18a0b"}
Oct 30 19:20:17.363185 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:17.363129    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n" podStartSLOduration=3.337604004 podStartE2EDuration="11.363112137s" totalImagesPullingTime="8.025508133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:20:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:17.362704223 +0000 UTC m=+11.847360334" watchObservedRunningTime="2026-10-30 19:20:17.363112137 +0000 UTC m=+11.847768248"
Oct 30 19:20:17.931215 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:17.931177    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l65sf"
Oct 30 19:20:17.931970 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:17.931793    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l65sf"
Oct 30 19:20:18.099519 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:18.099463    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:18.099519 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:18.099502    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:18.099677 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:18.099614    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcr5t" podUID="029fdc09-aaca-4dc2-90ff-86e05c26bf9e"
Oct 30 19:20:18.099790 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:18.099759    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnzx" podUID="c1a81d1f-b828-4a0d-a57a-df3d493947d5"
Oct 30 19:20:18.282091 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:18.282056    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2gf9g" event={"ID":"c4aa3418-bbc0-4c70-a42a-2a65e478b47d","Type":"ContainerStarted","Data":"13994616485b4a1fad817e5ce1ed455aae57a521e7b93d03969f7e4500d98ff2"}
Oct 30 19:20:18.285155 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:18.285122    2582 generic.go:356] "Generic (PLEG): container finished" podID="aacc7bb9-0109-4984-884f-50f8d2144559" containerID="276381a3abdd34d4a03e249c861ef6b23c7c296777f471f62a1cbc2915af23e8" exitCode=0
Oct 30 19:20:18.285282 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:18.285164    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jjm2b" event={"ID":"aacc7bb9-0109-4984-884f-50f8d2144559","Type":"ContainerDied","Data":"276381a3abdd34d4a03e249c861ef6b23c7c296777f471f62a1cbc2915af23e8"}
Oct 30 19:20:18.287087 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:18.287060    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lqm86" event={"ID":"f29fd70f-b10e-4154-8652-6b17fbe3cf13","Type":"ContainerStarted","Data":"089243a273d68d9e6ebeb50ce7bdbf478a506d8e673d4d434db5a79a7d775ec5"}
Oct 30 19:20:18.321277 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:18.321233    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2gf9g" podStartSLOduration=2.585581858 podStartE2EDuration="12.321219979s" totalImagesPullingTime="9.735638121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:18.320924026 +0000 UTC m=+12.805580138" watchObservedRunningTime="2026-10-30 19:20:18.321219979 +0000 UTC m=+12.805876091"
Oct 30 19:20:18.408174 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:18.408116    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lqm86" podStartSLOduration=4.3879651729999996 podStartE2EDuration="12.408097649s" totalImagesPullingTime="8.020132476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:18.407453342 +0000 UTC m=+12.892109454" watchObservedRunningTime="2026-10-30 19:20:18.408097649 +0000 UTC m=+12.892753761"
Oct 30 19:20:19.100221 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:19.100181    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:19.100412 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:19.100310    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27xsq" podUID="bbd278f9-76f5-41ed-93d1-435804741834"
Oct 30 19:20:20.099499 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:20.099235    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:20.099499 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:20.099274    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:20.099499 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:20.099388    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcr5t" podUID="029fdc09-aaca-4dc2-90ff-86e05c26bf9e"
Oct 30 19:20:20.099499 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:20.099470    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnzx" podUID="c1a81d1f-b828-4a0d-a57a-df3d493947d5"
Oct 30 19:20:20.294552 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:20.294527    2582 generic.go:356] "Generic (PLEG): container finished" podID="aacc7bb9-0109-4984-884f-50f8d2144559" containerID="602b3ffee49b831edd1d0c8f8983c3f9f8b013766c73806ccfa51aad09b6e8f7" exitCode=0
Oct 30 19:20:20.294652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:20.294574    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jjm2b" event={"ID":"aacc7bb9-0109-4984-884f-50f8d2144559","Type":"ContainerDied","Data":"602b3ffee49b831edd1d0c8f8983c3f9f8b013766c73806ccfa51aad09b6e8f7"}
Oct 30 19:20:21.099994 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:21.099757    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:21.100383 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:21.100091    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27xsq" podUID="bbd278f9-76f5-41ed-93d1-435804741834"
Oct 30 19:20:21.299797 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:21.299763    2582 generic.go:356] "Generic (PLEG): container finished" podID="aacc7bb9-0109-4984-884f-50f8d2144559" containerID="ee6ebeef62c630742d85a28264ede1ca43056ddd690995a3ef874e5c2b9a7852" exitCode=0
Oct 30 19:20:21.299961 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:21.299812    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jjm2b" event={"ID":"aacc7bb9-0109-4984-884f-50f8d2144559","Type":"ContainerDied","Data":"ee6ebeef62c630742d85a28264ede1ca43056ddd690995a3ef874e5c2b9a7852"}
Oct 30 19:20:21.897736 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:21.897704    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l65sf"
Oct 30 19:20:21.897908 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:21.897830    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:20:21.898627 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:21.898580    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l65sf"
Oct 30 19:20:22.099698 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:22.099655    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:22.099698 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:22.099708    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:22.099919 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:22.099795    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcr5t" podUID="029fdc09-aaca-4dc2-90ff-86e05c26bf9e"
Oct 30 19:20:22.099972 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:22.099923    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnzx" podUID="c1a81d1f-b828-4a0d-a57a-df3d493947d5"
Oct 30 19:20:22.305517 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:22.305438    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jjm2b" event={"ID":"aacc7bb9-0109-4984-884f-50f8d2144559","Type":"ContainerStarted","Data":"e896370f492fb87203b21a877de8c6a6d217866aba0979ed9cd82fd044dcae1d"}
Oct 30 19:20:23.099533 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:23.099356    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:23.099668 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:23.099602    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27xsq" podUID="bbd278f9-76f5-41ed-93d1-435804741834"
Oct 30 19:20:23.309703 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:23.309642    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:20:23.310062 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:23.309970    2582 generic.go:356] "Generic (PLEG): container finished" podID="8c299f86-7f6e-4666-ab9c-542537e6ff8e" containerID="98329262d3f3ece16ef013d60d3b780cb5a91c7ef6ef9056998c63c47fdef0b0" exitCode=1
Oct 30 19:20:23.310062 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:23.310039    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc" event={"ID":"8c299f86-7f6e-4666-ab9c-542537e6ff8e","Type":"ContainerStarted","Data":"5bd6a5f7b3a47d6870c3592d1445a6a863ee876ea9e9bfe532a9e52838aec29a"}
Oct 30 19:20:23.310132 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:23.310094    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc" event={"ID":"8c299f86-7f6e-4666-ab9c-542537e6ff8e","Type":"ContainerStarted","Data":"1a3d67dbd6b0625b863ecd833f8b95409b15fcb38a22ae83d76d2ca59c2999c1"}
Oct 30 19:20:23.310132 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:23.310123    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc" event={"ID":"8c299f86-7f6e-4666-ab9c-542537e6ff8e","Type":"ContainerStarted","Data":"4ef67ba1b72be6b764ee0c0e4183149a045c3be5838590ecf7e1831fd737f299"}
Oct 30 19:20:23.310188 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:23.310152    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc" event={"ID":"8c299f86-7f6e-4666-ab9c-542537e6ff8e","Type":"ContainerStarted","Data":"e95d18b29e587a8476a0bb6698ea9608f51cd92f613c66722049bc1dc80da9dc"}
Oct 30 19:20:23.310188 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:23.310178    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc" event={"ID":"8c299f86-7f6e-4666-ab9c-542537e6ff8e","Type":"ContainerDied","Data":"98329262d3f3ece16ef013d60d3b780cb5a91c7ef6ef9056998c63c47fdef0b0"}
Oct 30 19:20:23.310243 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:23.310206    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc" event={"ID":"8c299f86-7f6e-4666-ab9c-542537e6ff8e","Type":"ContainerStarted","Data":"4735ff12ef190bc3c9263eb1dd9ef473b54d97d0b44250c54bf513f80df3ad55"}
Oct 30 19:20:23.715289 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:23.715264    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs\") pod \"network-metrics-daemon-vcr5t\" (UID: \"029fdc09-aaca-4dc2-90ff-86e05c26bf9e\") " pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:23.715439 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:23.715294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:23.715439 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:23.715409    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:20:23.715518 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:23.715449    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:20:23.715518 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:23.715463    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret podName:bbd278f9-76f5-41ed-93d1-435804741834 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:39.715445106 +0000 UTC m=+34.200101215 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret") pod "global-pull-secret-syncer-27xsq" (UID: "bbd278f9-76f5-41ed-93d1-435804741834") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:20:23.715518 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:23.715482    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs podName:029fdc09-aaca-4dc2-90ff-86e05c26bf9e nodeName:}" failed. No retries permitted until 2026-10-30 19:20:39.715472632 +0000 UTC m=+34.200128730 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs") pod "network-metrics-daemon-vcr5t" (UID: "029fdc09-aaca-4dc2-90ff-86e05c26bf9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:20:23.815922 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:23.815893    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9zdg\" (UniqueName: \"kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg\") pod \"network-check-target-6dnzx\" (UID: \"c1a81d1f-b828-4a0d-a57a-df3d493947d5\") " pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:23.816062 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:23.815991    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:20:23.816062 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:23.816006    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:20:23.816062 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:23.816014    2582 projected.go:196] Error preparing data for projected volume kube-api-access-r9zdg for pod openshift-network-diagnostics/network-check-target-6dnzx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:20:23.816062 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:23.816055    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg podName:c1a81d1f-b828-4a0d-a57a-df3d493947d5 nodeName:}" failed. No retries permitted until 2026-10-30 19:20:39.816043579 +0000 UTC m=+34.300699668 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r9zdg" (UniqueName: "kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg") pod "network-check-target-6dnzx" (UID: "c1a81d1f-b828-4a0d-a57a-df3d493947d5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:20:24.099405 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:24.099311    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:24.099552 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:24.099316    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:24.099552 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:24.099434    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcr5t" podUID="029fdc09-aaca-4dc2-90ff-86e05c26bf9e"
Oct 30 19:20:24.099552 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:24.099507    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnzx" podUID="c1a81d1f-b828-4a0d-a57a-df3d493947d5"
Oct 30 19:20:25.099888 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:25.099866    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:25.100214 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:25.099953    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27xsq" podUID="bbd278f9-76f5-41ed-93d1-435804741834"
Oct 30 19:20:25.316878 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:25.316805    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:20:25.317196 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:25.317177    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc" event={"ID":"8c299f86-7f6e-4666-ab9c-542537e6ff8e","Type":"ContainerStarted","Data":"f16094b35c59a8f4186f031882a0541a8aee787a349888c64d080bae239c4a47"}
Oct 30 19:20:26.100718 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:26.100686    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:26.101184 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:26.100767    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcr5t" podUID="029fdc09-aaca-4dc2-90ff-86e05c26bf9e"
Oct 30 19:20:26.101184 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:26.100870    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:26.101184 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:26.100979    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnzx" podUID="c1a81d1f-b828-4a0d-a57a-df3d493947d5"
Oct 30 19:20:27.099248 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:27.099217    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:27.099417 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:27.099302    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27xsq" podUID="bbd278f9-76f5-41ed-93d1-435804741834"
Oct 30 19:20:27.325416 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:27.325394    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:20:27.326017 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:27.325769    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc" event={"ID":"8c299f86-7f6e-4666-ab9c-542537e6ff8e","Type":"ContainerStarted","Data":"d2eabeec3c94a77c87a3faba8822dd39f8167c086909e6a7f68654343cc1f093"}
Oct 30 19:20:27.326017 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:27.326003    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:27.326356 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:27.326340    2582 scope.go:122] "RemoveContainer" containerID="98329262d3f3ece16ef013d60d3b780cb5a91c7ef6ef9056998c63c47fdef0b0"
Oct 30 19:20:27.349428 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:27.346047    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:27.412353 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:27.412305    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jjm2b" podStartSLOduration=11.314041371 podStartE2EDuration="21.412294184s" totalImagesPullingTime="10.098252813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:20:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:22.351291568 +0000 UTC m=+16.835947690" watchObservedRunningTime="2026-10-30 19:20:27.412294184 +0000 UTC m=+21.896950295"
Oct 30 19:20:27.529589 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:27.529552    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:28.099154 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:28.099119    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:28.099154 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:28.099147    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:28.099348 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:28.099239    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnzx" podUID="c1a81d1f-b828-4a0d-a57a-df3d493947d5"
Oct 30 19:20:28.099348 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:28.099325    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcr5t" podUID="029fdc09-aaca-4dc2-90ff-86e05c26bf9e"
Oct 30 19:20:28.331052 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:28.331031    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:20:28.331434 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:28.331408    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc" event={"ID":"8c299f86-7f6e-4666-ab9c-542537e6ff8e","Type":"ContainerStarted","Data":"c7aaa37230e23a0419d5f94fc51255840481a067a95a6008137efdb4a9402872"}
Oct 30 19:20:28.331783 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:28.331759    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:28.349752 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:28.349682    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:28.433505 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:28.433450    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc" podStartSLOduration=7.977009742 podStartE2EDuration="22.433428209s" totalImagesPullingTime="14.456418467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:28.432772705 +0000 UTC m=+22.917428827" watchObservedRunningTime="2026-10-30 19:20:28.433428209 +0000 UTC m=+22.918084323"
Oct 30 19:20:29.099915 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:29.099888    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:29.100071 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:29.100002    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27xsq" podUID="bbd278f9-76f5-41ed-93d1-435804741834"
Oct 30 19:20:29.256548 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:29.256479    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6dnzx"]
Oct 30 19:20:29.256691 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:29.256577    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:29.256691 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:29.256658    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnzx" podUID="c1a81d1f-b828-4a0d-a57a-df3d493947d5"
Oct 30 19:20:29.260833 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:29.260809    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-27xsq"]
Oct 30 19:20:29.261300 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:29.261282    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vcr5t"]
Oct 30 19:20:29.261392 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:29.261380    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:29.261850 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:29.261822    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcr5t" podUID="029fdc09-aaca-4dc2-90ff-86e05c26bf9e"
Oct 30 19:20:29.333728 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:29.333701    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:29.334216 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:29.333787    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27xsq" podUID="bbd278f9-76f5-41ed-93d1-435804741834"
Oct 30 19:20:29.359373 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:29.359342    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:29.543814 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:29.543740    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:29.744948 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:29.744917    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:31.099539 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:31.099513    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:31.100066 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:31.099581    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:31.100066 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:31.099605    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:31.100066 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:31.099704    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27xsq" podUID="bbd278f9-76f5-41ed-93d1-435804741834"
Oct 30 19:20:31.100066 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:31.099729    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnzx" podUID="c1a81d1f-b828-4a0d-a57a-df3d493947d5"
Oct 30 19:20:31.100066 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:31.099811    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcr5t" podUID="029fdc09-aaca-4dc2-90ff-86e05c26bf9e"
Oct 30 19:20:32.039192 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:32.039107    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:32.190305 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:32.190274    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:33.100132 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:33.100105    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:33.100345 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:33.100137    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:33.100345 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:33.100242    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:33.100345 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:33.100253    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcr5t" podUID="029fdc09-aaca-4dc2-90ff-86e05c26bf9e"
Oct 30 19:20:33.100345 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:33.100313    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27xsq" podUID="bbd278f9-76f5-41ed-93d1-435804741834"
Oct 30 19:20:33.100507 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:33.100379    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnzx" podUID="c1a81d1f-b828-4a0d-a57a-df3d493947d5"
Oct 30 19:20:35.099279 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.099249    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:35.099279 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.099286    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:35.099784 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.099282    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:35.099784 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:35.099353    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6dnzx" podUID="c1a81d1f-b828-4a0d-a57a-df3d493947d5"
Oct 30 19:20:35.099784 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:35.099456    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-27xsq" podUID="bbd278f9-76f5-41ed-93d1-435804741834"
Oct 30 19:20:35.099784 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:35.099521    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vcr5t" podUID="029fdc09-aaca-4dc2-90ff-86e05c26bf9e"
Oct 30 19:20:35.414356 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.414183    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:35.574555 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.574528    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:35.777927 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.777871    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeReady"
Oct 30 19:20:35.778045 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.778000    2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 19:20:35.835331 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.835308    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bz68x"]
Oct 30 19:20:35.888832 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.888809    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bm87v"]
Oct 30 19:20:35.888983 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.888900    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bz68x"
Oct 30 19:20:35.902104 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.902086    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bz68x"]
Oct 30 19:20:35.902183 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.902117    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bm87v"]
Oct 30 19:20:35.902220 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.902189    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:35.904580 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.904559    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:20:35.904677 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.904637    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:20:35.938018 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.938002    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lkdbb\""
Oct 30 19:20:35.939189 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.939169    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:20:35.939189 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.939179    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:20:35.939321 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.939313    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m2qt2\""
Oct 30 19:20:35.966035 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.966016    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:20:35.977268 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:35.977239    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:36.007332 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.007307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b368ff6-93f5-472f-97ae-3f9ffee9102d-tmp-dir\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:36.007434 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.007341    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8cf3812-0483-481a-86e6-8c20c76d3426-cert\") pod \"ingress-canary-bz68x\" (UID: \"e8cf3812-0483-481a-86e6-8c20c76d3426\") " pod="openshift-ingress-canary/ingress-canary-bz68x"
Oct 30 19:20:36.007478 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.007446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwzm8\" (UniqueName: \"kubernetes.io/projected/1b368ff6-93f5-472f-97ae-3f9ffee9102d-kube-api-access-nwzm8\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:36.007518 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.007489    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b368ff6-93f5-472f-97ae-3f9ffee9102d-metrics-tls\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:36.007617 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.007601    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk84z\" (UniqueName: \"kubernetes.io/projected/e8cf3812-0483-481a-86e6-8c20c76d3426-kube-api-access-hk84z\") pod \"ingress-canary-bz68x\" (UID: \"e8cf3812-0483-481a-86e6-8c20c76d3426\") " pod="openshift-ingress-canary/ingress-canary-bz68x"
Oct 30 19:20:36.007663 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.007631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b368ff6-93f5-472f-97ae-3f9ffee9102d-config-volume\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:36.109880 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.107820    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b368ff6-93f5-472f-97ae-3f9ffee9102d-tmp-dir\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:36.109880 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.108239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8cf3812-0483-481a-86e6-8c20c76d3426-cert\") pod \"ingress-canary-bz68x\" (UID: \"e8cf3812-0483-481a-86e6-8c20c76d3426\") " pod="openshift-ingress-canary/ingress-canary-bz68x"
Oct 30 19:20:36.109880 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.108382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwzm8\" (UniqueName: \"kubernetes.io/projected/1b368ff6-93f5-472f-97ae-3f9ffee9102d-kube-api-access-nwzm8\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:36.109880 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.108448    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b368ff6-93f5-472f-97ae-3f9ffee9102d-tmp-dir\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:36.109880 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.108453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b368ff6-93f5-472f-97ae-3f9ffee9102d-metrics-tls\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:36.109880 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:36.108697    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:20:36.109880 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.108812    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk84z\" (UniqueName: \"kubernetes.io/projected/e8cf3812-0483-481a-86e6-8c20c76d3426-kube-api-access-hk84z\") pod \"ingress-canary-bz68x\" (UID: \"e8cf3812-0483-481a-86e6-8c20c76d3426\") " pod="openshift-ingress-canary/ingress-canary-bz68x"
Oct 30 19:20:36.109880 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.108850    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b368ff6-93f5-472f-97ae-3f9ffee9102d-config-volume\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:36.109880 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:36.108958    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b368ff6-93f5-472f-97ae-3f9ffee9102d-metrics-tls podName:1b368ff6-93f5-472f-97ae-3f9ffee9102d nodeName:}" failed. No retries permitted until 2026-10-30 19:20:36.608844098 +0000 UTC m=+31.093500187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b368ff6-93f5-472f-97ae-3f9ffee9102d-metrics-tls") pod "dns-default-bm87v" (UID: "1b368ff6-93f5-472f-97ae-3f9ffee9102d") : secret "dns-default-metrics-tls" not found
Oct 30 19:20:36.110689 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.110171    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b368ff6-93f5-472f-97ae-3f9ffee9102d-config-volume\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:36.111242 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.111172    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nxtzp"]
Oct 30 19:20:36.114558 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.114537    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8cf3812-0483-481a-86e6-8c20c76d3426-cert\") pod \"ingress-canary-bz68x\" (UID: \"e8cf3812-0483-481a-86e6-8c20c76d3426\") " pod="openshift-ingress-canary/ingress-canary-bz68x"
Oct 30 19:20:36.137183 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.137159    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk84z\" (UniqueName: \"kubernetes.io/projected/e8cf3812-0483-481a-86e6-8c20c76d3426-kube-api-access-hk84z\") pod \"ingress-canary-bz68x\" (UID: \"e8cf3812-0483-481a-86e6-8c20c76d3426\") " pod="openshift-ingress-canary/ingress-canary-bz68x"
Oct 30 19:20:36.137445 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.137430    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwzm8\" (UniqueName: \"kubernetes.io/projected/1b368ff6-93f5-472f-97ae-3f9ffee9102d-kube-api-access-nwzm8\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:36.145772 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.145754    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nxtzp"]
Oct 30 19:20:36.145850 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.145840    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxtzp"
Oct 30 19:20:36.163508 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.163483    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:36.210073 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.210048    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d98ps\" (UniqueName: \"kubernetes.io/projected/7948dcd5-eb52-4510-a73d-25b869e1394e-kube-api-access-d98ps\") pod \"migrator-66df5d496b-nxtzp\" (UID: \"7948dcd5-eb52-4510-a73d-25b869e1394e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxtzp"
Oct 30 19:20:36.210990 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.210974    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bz68x"
Oct 30 19:20:36.256075 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.256055    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5ffd5\""
Oct 30 19:20:36.257610 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.257563    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 19:20:36.257781 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.257766    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 19:20:36.266677 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.266658    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dxp6q"]
Oct 30 19:20:36.282251 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.282231    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:36.294279 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.294260    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dxp6q"]
Oct 30 19:20:36.304250 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.304228    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-snzrt-w5xjz"]
Oct 30 19:20:36.310869 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.310837    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d98ps\" (UniqueName: \"kubernetes.io/projected/7948dcd5-eb52-4510-a73d-25b869e1394e-kube-api-access-d98ps\") pod \"migrator-66df5d496b-nxtzp\" (UID: \"7948dcd5-eb52-4510-a73d-25b869e1394e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxtzp"
Oct 30 19:20:36.313132 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.313114    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:36.326885 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.326865    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-snzrt-w5xjz"]
Oct 30 19:20:36.344930 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.344910    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d98ps\" (UniqueName: \"kubernetes.io/projected/7948dcd5-eb52-4510-a73d-25b869e1394e-kube-api-access-d98ps\") pod \"migrator-66df5d496b-nxtzp\" (UID: \"7948dcd5-eb52-4510-a73d-25b869e1394e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxtzp"
Oct 30 19:20:36.353318 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.353294    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:36.355595 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.355577    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:20:36.355595 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.355583    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qmnn8\""
Oct 30 19:20:36.355743 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.355729    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hh67w\""
Oct 30 19:20:36.358570 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.358556    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:20:36.358730 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.358714    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:20:36.358791 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.358779    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 19:20:36.371656 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.371633    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:20:36.372133 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.372119    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 19:20:36.412141 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.412119    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:36.412231 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.412145    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-service-ca-bundle\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:36.412231 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.412188    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cbf67380-1538-474c-ad65-4f8194a3d0af-crio-socket\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:36.412322 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.412244    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbf67380-1538-474c-ad65-4f8194a3d0af-data-volume\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:36.412396 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.412321    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbf67380-1538-474c-ad65-4f8194a3d0af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:36.412396 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.412385    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdmgh\" (UniqueName: \"kubernetes.io/projected/9838f46c-794d-4b6f-9f2d-a1a45e21819f-kube-api-access-kdmgh\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:36.412497 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.412475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9838f46c-794d-4b6f-9f2d-a1a45e21819f-archives-path\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:36.412539 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.412495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-trusted-ca-bundle\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:36.412539 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.412525    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9txd\" (UniqueName: \"kubernetes.io/projected/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-api-access-s9txd\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:36.478291 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.478272    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxtzp"
Oct 30 19:20:36.503271 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.503254    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-c49qz"]
Oct 30 19:20:36.513767 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.513747    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-trusted-ca-bundle\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:36.513833 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.513803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9txd\" (UniqueName: \"kubernetes.io/projected/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-api-access-s9txd\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:36.513892 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.513864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:36.513944 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.513892    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-service-ca-bundle\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:36.514016 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.513957    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cbf67380-1538-474c-ad65-4f8194a3d0af-crio-socket\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:36.514016 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:36.513969    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:20:36.514016 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.513985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbf67380-1538-474c-ad65-4f8194a3d0af-data-volume\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:36.514175 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.514014    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbf67380-1538-474c-ad65-4f8194a3d0af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:36.514175 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:36.514047    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-rbac-proxy-cm podName:cbf67380-1538-474c-ad65-4f8194a3d0af nodeName:}" failed. No retries permitted until 2026-10-30 19:20:37.014028676 +0000 UTC m=+31.498684787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dxp6q" (UID: "cbf67380-1538-474c-ad65-4f8194a3d0af") : configmap "kube-rbac-proxy" not found
Oct 30 19:20:36.514175 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.514069    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdmgh\" (UniqueName: \"kubernetes.io/projected/9838f46c-794d-4b6f-9f2d-a1a45e21819f-kube-api-access-kdmgh\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:36.514175 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:36.514109    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:20:36.514175 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.514109    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9838f46c-794d-4b6f-9f2d-a1a45e21819f-archives-path\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:36.514175 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.514169    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cbf67380-1538-474c-ad65-4f8194a3d0af-crio-socket\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:36.514578 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:36.514176    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbf67380-1538-474c-ad65-4f8194a3d0af-insights-runtime-extractor-tls podName:cbf67380-1538-474c-ad65-4f8194a3d0af nodeName:}" failed. No retries permitted until 2026-10-30 19:20:37.014143103 +0000 UTC m=+31.498799217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cbf67380-1538-474c-ad65-4f8194a3d0af-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dxp6q" (UID: "cbf67380-1538-474c-ad65-4f8194a3d0af") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:20:36.514578 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.514347    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbf67380-1538-474c-ad65-4f8194a3d0af-data-volume\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:36.514578 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.514417    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9838f46c-794d-4b6f-9f2d-a1a45e21819f-archives-path\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:36.514578 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.514494    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-service-ca-bundle\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:36.514836 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.514816    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-trusted-ca-bundle\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:36.524999 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.524977    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz"
Oct 30 19:20:36.526070 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.526046    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-c49qz"]
Oct 30 19:20:36.527786 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.527763    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:36.533382 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.533340    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9txd\" (UniqueName: \"kubernetes.io/projected/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-api-access-s9txd\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:36.534397 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.534347    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdmgh\" (UniqueName: \"kubernetes.io/projected/9838f46c-794d-4b6f-9f2d-a1a45e21819f-kube-api-access-kdmgh\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:36.537442 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.537424    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bz68x"]
Oct 30 19:20:36.539211 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:36.539192    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8cf3812_0483_481a_86e6_8c20c76d3426.slice/crio-ce51e3b3e7c2408cc012df9a5512716cfba5ea80bc2f9aafd8f72ad0130a320f WatchSource:0}: Error finding container ce51e3b3e7c2408cc012df9a5512716cfba5ea80bc2f9aafd8f72ad0130a320f: Status 404 returned error can't find the container with id ce51e3b3e7c2408cc012df9a5512716cfba5ea80bc2f9aafd8f72ad0130a320f
Oct 30 19:20:36.543843 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.542574    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 19:20:36.543843 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.542691    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-brmvf\""
Oct 30 19:20:36.543843 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.542757    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 19:20:36.543843 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.542836    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 19:20:36.543843 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.542892    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 19:20:36.615202 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.615179    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0cffcb83-c327-45ab-90b7-837f3f455f92-signing-cabundle\") pod \"service-ca-5fffc97bdf-c49qz\" (UID: \"0cffcb83-c327-45ab-90b7-837f3f455f92\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz"
Oct 30 19:20:36.615349 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.615330    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ll8f\" (UniqueName: \"kubernetes.io/projected/0cffcb83-c327-45ab-90b7-837f3f455f92-kube-api-access-5ll8f\") pod \"service-ca-5fffc97bdf-c49qz\" (UID: \"0cffcb83-c327-45ab-90b7-837f3f455f92\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz"
Oct 30 19:20:36.615414 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.615372    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0cffcb83-c327-45ab-90b7-837f3f455f92-signing-key\") pod \"service-ca-5fffc97bdf-c49qz\" (UID: \"0cffcb83-c327-45ab-90b7-837f3f455f92\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz"
Oct 30 19:20:36.615414 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.615395    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b368ff6-93f5-472f-97ae-3f9ffee9102d-metrics-tls\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:36.615491 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:36.615474    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:20:36.615527 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:36.615512    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b368ff6-93f5-472f-97ae-3f9ffee9102d-metrics-tls podName:1b368ff6-93f5-472f-97ae-3f9ffee9102d nodeName:}" failed. No retries permitted until 2026-10-30 19:20:37.615500018 +0000 UTC m=+32.100156110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b368ff6-93f5-472f-97ae-3f9ffee9102d-metrics-tls") pod "dns-default-bm87v" (UID: "1b368ff6-93f5-472f-97ae-3f9ffee9102d") : secret "dns-default-metrics-tls" not found
Oct 30 19:20:36.626186 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.626148    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:36.715947 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.715917    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0cffcb83-c327-45ab-90b7-837f3f455f92-signing-key\") pod \"service-ca-5fffc97bdf-c49qz\" (UID: \"0cffcb83-c327-45ab-90b7-837f3f455f92\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz"
Oct 30 19:20:36.716069 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.716050    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0cffcb83-c327-45ab-90b7-837f3f455f92-signing-cabundle\") pod \"service-ca-5fffc97bdf-c49qz\" (UID: \"0cffcb83-c327-45ab-90b7-837f3f455f92\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz"
Oct 30 19:20:36.716177 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.716161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ll8f\" (UniqueName: \"kubernetes.io/projected/0cffcb83-c327-45ab-90b7-837f3f455f92-kube-api-access-5ll8f\") pod \"service-ca-5fffc97bdf-c49qz\" (UID: \"0cffcb83-c327-45ab-90b7-837f3f455f92\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz"
Oct 30 19:20:36.719060 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.717234    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0cffcb83-c327-45ab-90b7-837f3f455f92-signing-cabundle\") pod \"service-ca-5fffc97bdf-c49qz\" (UID: \"0cffcb83-c327-45ab-90b7-837f3f455f92\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz"
Oct 30 19:20:36.719060 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.718929    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0cffcb83-c327-45ab-90b7-837f3f455f92-signing-key\") pod \"service-ca-5fffc97bdf-c49qz\" (UID: \"0cffcb83-c327-45ab-90b7-837f3f455f92\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz"
Oct 30 19:20:36.736277 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.736255    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ll8f\" (UniqueName: \"kubernetes.io/projected/0cffcb83-c327-45ab-90b7-837f3f455f92-kube-api-access-5ll8f\") pod \"service-ca-5fffc97bdf-c49qz\" (UID: \"0cffcb83-c327-45ab-90b7-837f3f455f92\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz"
Oct 30 19:20:36.797048 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.797030    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nxtzp"]
Oct 30 19:20:36.798680 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:36.798658    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7948dcd5_eb52_4510_a73d_25b869e1394e.slice/crio-731a90598c958a1bf0e7c98cef4b5f80c910a4e391bd5501f60dd005931f49ad WatchSource:0}: Error finding container 731a90598c958a1bf0e7c98cef4b5f80c910a4e391bd5501f60dd005931f49ad: Status 404 returned error can't find the container with id 731a90598c958a1bf0e7c98cef4b5f80c910a4e391bd5501f60dd005931f49ad
Oct 30 19:20:36.866611 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.866589    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz"
Oct 30 19:20:36.889996 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.889937    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:36.941787 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:36.941752    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-snzrt-w5xjz"]
Oct 30 19:20:36.944466 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:36.944436    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9838f46c_794d_4b6f_9f2d_a1a45e21819f.slice/crio-b1fdf1c7824e8af0757de524fb1b5161ca4c06bd4777aadccf3d7c78419cbd60 WatchSource:0}: Error finding container b1fdf1c7824e8af0757de524fb1b5161ca4c06bd4777aadccf3d7c78419cbd60: Status 404 returned error can't find the container with id b1fdf1c7824e8af0757de524fb1b5161ca4c06bd4777aadccf3d7c78419cbd60
Oct 30 19:20:37.018623 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.018589    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbf67380-1538-474c-ad65-4f8194a3d0af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:37.018799 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.018661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:37.018799 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:37.018749    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:20:37.018799 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:37.018755    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:20:37.018928 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:37.018808    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-rbac-proxy-cm podName:cbf67380-1538-474c-ad65-4f8194a3d0af nodeName:}" failed. No retries permitted until 2026-10-30 19:20:38.018795075 +0000 UTC m=+32.503451169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dxp6q" (UID: "cbf67380-1538-474c-ad65-4f8194a3d0af") : configmap "kube-rbac-proxy" not found
Oct 30 19:20:37.018928 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:37.018820    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbf67380-1538-474c-ad65-4f8194a3d0af-insights-runtime-extractor-tls podName:cbf67380-1538-474c-ad65-4f8194a3d0af nodeName:}" failed. No retries permitted until 2026-10-30 19:20:38.018814779 +0000 UTC m=+32.503470867 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cbf67380-1538-474c-ad65-4f8194a3d0af-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dxp6q" (UID: "cbf67380-1538-474c-ad65-4f8194a3d0af") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:20:37.085229 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.085196    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:37.100053 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.100029    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:37.100162 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.100060    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:37.100216 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.100205    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:37.116948 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.116924    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:20:37.117507 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.117398    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:20:37.117507 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.117416    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:20:37.117507 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.117421    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sl59z\""
Oct 30 19:20:37.117507 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.117429    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:20:37.117507 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.117475    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-77hs9\""
Oct 30 19:20:37.117742 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.117550    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-65nsn\""
Oct 30 19:20:37.185240 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.185179    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-c49qz"]
Oct 30 19:20:37.187945 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:37.187919    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cffcb83_c327_45ab_90b7_837f3f455f92.slice/crio-58ef30be00e42d2ac86281d2a108867103e940ebaec4fcf815e1bd92d333c82e WatchSource:0}: Error finding container 58ef30be00e42d2ac86281d2a108867103e940ebaec4fcf815e1bd92d333c82e: Status 404 returned error can't find the container with id 58ef30be00e42d2ac86281d2a108867103e940ebaec4fcf815e1bd92d333c82e
Oct 30 19:20:37.351176 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.351142    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxtzp" event={"ID":"7948dcd5-eb52-4510-a73d-25b869e1394e","Type":"ContainerStarted","Data":"731a90598c958a1bf0e7c98cef4b5f80c910a4e391bd5501f60dd005931f49ad"}
Oct 30 19:20:37.352416 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.352392    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz" event={"ID":"0cffcb83-c327-45ab-90b7-837f3f455f92","Type":"ContainerStarted","Data":"58ef30be00e42d2ac86281d2a108867103e940ebaec4fcf815e1bd92d333c82e"}
Oct 30 19:20:37.353652 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.353630    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bz68x" event={"ID":"e8cf3812-0483-481a-86e6-8c20c76d3426","Type":"ContainerStarted","Data":"ce51e3b3e7c2408cc012df9a5512716cfba5ea80bc2f9aafd8f72ad0130a320f"}
Oct 30 19:20:37.354725 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.354707    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-snzrt-w5xjz" event={"ID":"9838f46c-794d-4b6f-9f2d-a1a45e21819f","Type":"ContainerStarted","Data":"b1fdf1c7824e8af0757de524fb1b5161ca4c06bd4777aadccf3d7c78419cbd60"}
Oct 30 19:20:37.624251 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.624213    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b368ff6-93f5-472f-97ae-3f9ffee9102d-metrics-tls\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:37.624427 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:37.624403    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:20:37.624508 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:37.624482    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b368ff6-93f5-472f-97ae-3f9ffee9102d-metrics-tls podName:1b368ff6-93f5-472f-97ae-3f9ffee9102d nodeName:}" failed. No retries permitted until 2026-10-30 19:20:39.62446173 +0000 UTC m=+34.109117831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b368ff6-93f5-472f-97ae-3f9ffee9102d-metrics-tls") pod "dns-default-bm87v" (UID: "1b368ff6-93f5-472f-97ae-3f9ffee9102d") : secret "dns-default-metrics-tls" not found
Oct 30 19:20:37.633242 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:37.633212    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:38.028395 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:38.028296    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbf67380-1538-474c-ad65-4f8194a3d0af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:38.028566 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:38.028424    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:38.028566 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:38.028521    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:20:38.028693 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:38.028566    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:20:38.028693 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:38.028582    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-rbac-proxy-cm podName:cbf67380-1538-474c-ad65-4f8194a3d0af nodeName:}" failed. No retries permitted until 2026-10-30 19:20:40.028564335 +0000 UTC m=+34.513220440 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dxp6q" (UID: "cbf67380-1538-474c-ad65-4f8194a3d0af") : configmap "kube-rbac-proxy" not found
Oct 30 19:20:38.028693 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:38.028631    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbf67380-1538-474c-ad65-4f8194a3d0af-insights-runtime-extractor-tls podName:cbf67380-1538-474c-ad65-4f8194a3d0af nodeName:}" failed. No retries permitted until 2026-10-30 19:20:40.028612976 +0000 UTC m=+34.513269078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cbf67380-1538-474c-ad65-4f8194a3d0af-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dxp6q" (UID: "cbf67380-1538-474c-ad65-4f8194a3d0af") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:20:39.363098 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:39.362658    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxtzp" event={"ID":"7948dcd5-eb52-4510-a73d-25b869e1394e","Type":"ContainerStarted","Data":"21f7924ab1efbf8089d585cac19cda5a3e2ea923036ea8592d04d25a0b0b7cd4"}
Oct 30 19:20:39.413451 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:39.413388    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxtzp" podStartSLOduration=1.546837401 podStartE2EDuration="3.413352306s" totalImagesPullingTime="1.866514905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:39.411134564 +0000 UTC m=+33.895790679" watchObservedRunningTime="2026-10-30 19:20:39.413352306 +0000 UTC m=+33.898008418"
Oct 30 19:20:39.643164 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:39.643122    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b368ff6-93f5-472f-97ae-3f9ffee9102d-metrics-tls\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:39.643351 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:39.643294    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:20:39.643486 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:39.643396    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b368ff6-93f5-472f-97ae-3f9ffee9102d-metrics-tls podName:1b368ff6-93f5-472f-97ae-3f9ffee9102d nodeName:}" failed. No retries permitted until 2026-10-30 19:20:43.643375694 +0000 UTC m=+38.128031808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b368ff6-93f5-472f-97ae-3f9ffee9102d-metrics-tls") pod "dns-default-bm87v" (UID: "1b368ff6-93f5-472f-97ae-3f9ffee9102d") : secret "dns-default-metrics-tls" not found
Oct 30 19:20:39.743788 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:39.743755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs\") pod \"network-metrics-daemon-vcr5t\" (UID: \"029fdc09-aaca-4dc2-90ff-86e05c26bf9e\") " pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:39.743788 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:39.743791    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:39.743978 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:39.743898    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:20:39.743978 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:39.743961    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs podName:029fdc09-aaca-4dc2-90ff-86e05c26bf9e nodeName:}" failed. No retries permitted until 2026-10-30 19:21:11.743945793 +0000 UTC m=+66.228601883 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs") pod "network-metrics-daemon-vcr5t" (UID: "029fdc09-aaca-4dc2-90ff-86e05c26bf9e") : secret "metrics-daemon-secret" not found
Oct 30 19:20:39.745886 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:39.745864    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:39.837574 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:39.837547    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:39.844834 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:39.844811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9zdg\" (UniqueName: \"kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg\") pod \"network-check-target-6dnzx\" (UID: \"c1a81d1f-b828-4a0d-a57a-df3d493947d5\") " pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:39.847437 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:39.847111    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9zdg\" (UniqueName: \"kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg\") pod \"network-check-target-6dnzx\" (UID: \"c1a81d1f-b828-4a0d-a57a-df3d493947d5\") " pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:39.852153 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:39.852136    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:39.997459 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:39.997355    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:40.046332 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:40.046310    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:40.046454 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:40.046383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbf67380-1538-474c-ad65-4f8194a3d0af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:40.046494 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:40.046485    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:20:40.046552 ip-10-0-140-248 kubenswrapper[2582]: E1030 19:20:40.046542    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbf67380-1538-474c-ad65-4f8194a3d0af-insights-runtime-extractor-tls podName:cbf67380-1538-474c-ad65-4f8194a3d0af nodeName:}" failed. No retries permitted until 2026-10-30 19:20:44.046524579 +0000 UTC m=+38.531180669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cbf67380-1538-474c-ad65-4f8194a3d0af-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dxp6q" (UID: "cbf67380-1538-474c-ad65-4f8194a3d0af") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:20:40.046756 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:40.046739    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:40.144424 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:40.144394    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-27xsq"]
Oct 30 19:20:40.157259 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:40.157233    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6dnzx"]
Oct 30 19:20:40.258452 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:40.258427    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd278f9_76f5_41ed_93d1_435804741834.slice/crio-fd650ca092f9d13b2e26a0ac96286f7cc347d013f8951d5e184bb33cbb2d503b WatchSource:0}: Error finding container fd650ca092f9d13b2e26a0ac96286f7cc347d013f8951d5e184bb33cbb2d503b: Status 404 returned error can't find the container with id fd650ca092f9d13b2e26a0ac96286f7cc347d013f8951d5e184bb33cbb2d503b
Oct 30 19:20:40.259094 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:40.259071    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a81d1f_b828_4a0d_a57a_df3d493947d5.slice/crio-c1e95132776f9a23e7fc3f5da5ac31b3e51af3c53f27b1574ff98a9d877f3db3 WatchSource:0}: Error finding container c1e95132776f9a23e7fc3f5da5ac31b3e51af3c53f27b1574ff98a9d877f3db3: Status 404 returned error can't find the container with id c1e95132776f9a23e7fc3f5da5ac31b3e51af3c53f27b1574ff98a9d877f3db3
Oct 30 19:20:40.365792 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:40.365769    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6dnzx" event={"ID":"c1a81d1f-b828-4a0d-a57a-df3d493947d5","Type":"ContainerStarted","Data":"c1e95132776f9a23e7fc3f5da5ac31b3e51af3c53f27b1574ff98a9d877f3db3"}
Oct 30 19:20:40.366831 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:40.366808    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-27xsq" event={"ID":"bbd278f9-76f5-41ed-93d1-435804741834","Type":"ContainerStarted","Data":"fd650ca092f9d13b2e26a0ac96286f7cc347d013f8951d5e184bb33cbb2d503b"}
Oct 30 19:20:41.372405 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:41.372150    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz" event={"ID":"0cffcb83-c327-45ab-90b7-837f3f455f92","Type":"ContainerStarted","Data":"50ffb68b147449e0da9d862f85ce5fbe6b3fdb01d334f08aec3ddc5f2ab9550a"}
Oct 30 19:20:41.375056 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:41.375003    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bz68x" event={"ID":"e8cf3812-0483-481a-86e6-8c20c76d3426","Type":"ContainerStarted","Data":"70f63890c1000cd7ce9923eeb1df7a649ff3c6a59c077e118eb332f7acfdd52a"}
Oct 30 19:20:41.376734 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:41.376706    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-snzrt-w5xjz" event={"ID":"9838f46c-794d-4b6f-9f2d-a1a45e21819f","Type":"ContainerStarted","Data":"99f15990511a6c5f7860ba9ce9fdf6cc29aac7ed1eaf3bb5475d30d41e764b6c"}
Oct 30 19:20:41.413881 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:41.413821    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz" podStartSLOduration=2.090913079 podStartE2EDuration="5.413807858s" totalImagesPullingTime="3.322894779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:41.412717715 +0000 UTC m=+35.897373830" watchObservedRunningTime="2026-10-30 19:20:41.413807858 +0000 UTC m=+35.898463970"
Oct 30 19:20:41.452631 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:41.452586    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-snzrt-w5xjz" podStartSLOduration=1.901470414 podStartE2EDuration="5.452575s" totalImagesPullingTime="3.551104586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:41.451719065 +0000 UTC m=+35.936375181" watchObservedRunningTime="2026-10-30 19:20:41.452575 +0000 UTC m=+35.937231156"
Oct 30 19:20:41.494788 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:41.494737    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bz68x" podStartSLOduration=2.538627285 podStartE2EDuration="6.49472134s" totalImagesPullingTime="3.956094055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:41.49365045 +0000 UTC m=+35.978306562" watchObservedRunningTime="2026-10-30 19:20:41.49472134 +0000 UTC m=+35.979377451"
Oct 30 19:20:41.595052 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:41.595018    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:41.756760 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:41.756683    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:41.946390 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:41.946342    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:42.340347 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:42.340263    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:43.103836 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:43.103800    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:43.249693 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:43.249653    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:43.491975 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:43.491940    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:43.527683 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:43.527657    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n9frl_90c7079a-e03a-4d3e-bb41-b61ce36292ec/dns-node-resolver/0.log"
Oct 30 19:20:43.640824 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:43.640769    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:43.679351 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:43.679314    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b368ff6-93f5-472f-97ae-3f9ffee9102d-metrics-tls\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:43.687138 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:43.687102    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b368ff6-93f5-472f-97ae-3f9ffee9102d-metrics-tls\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:43.736322 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:43.736285    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:43.785720 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:43.785634    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:43.991320 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:43.991286    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:44.084467 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:44.084396    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bm87v"]
Oct 30 19:20:44.085662 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:44.085635    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbf67380-1538-474c-ad65-4f8194a3d0af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:44.088242 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:44.088203    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbf67380-1538-474c-ad65-4f8194a3d0af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:44.088336 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:44.088268    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b368ff6_93f5_472f_97ae_3f9ffee9102d.slice/crio-41f07742d96fc257c2ba7769b5c4178e2a1b53570178cb198523caa4f19dbbc6 WatchSource:0}: Error finding container 41f07742d96fc257c2ba7769b5c4178e2a1b53570178cb198523caa4f19dbbc6: Status 404 returned error can't find the container with id 41f07742d96fc257c2ba7769b5c4178e2a1b53570178cb198523caa4f19dbbc6
Oct 30 19:20:44.113653 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:44.113629    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:44.206869 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:44.206832    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:44.326884 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:44.326858    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qrsz7_a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada/node-ca/0.log"
Oct 30 19:20:44.387062 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:44.387029    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bm87v" event={"ID":"1b368ff6-93f5-472f-97ae-3f9ffee9102d","Type":"ContainerStarted","Data":"41f07742d96fc257c2ba7769b5c4178e2a1b53570178cb198523caa4f19dbbc6"}
Oct 30 19:20:44.422839 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:44.422810    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dxp6q"]
Oct 30 19:20:44.475348 ip-10-0-140-248 kubenswrapper[2582]: W1030 19:20:44.475316    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf67380_1538_474c_ad65_4f8194a3d0af.slice/crio-0dcf5d06886b771a06a3cdcb7285942b84e1174f01dfad78de15b3691e852d03 WatchSource:0}: Error finding container 0dcf5d06886b771a06a3cdcb7285942b84e1174f01dfad78de15b3691e852d03: Status 404 returned error can't find the container with id 0dcf5d06886b771a06a3cdcb7285942b84e1174f01dfad78de15b3691e852d03
Oct 30 19:20:44.926914 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:44.926883    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bz68x_e8cf3812-0483-481a-86e6-8c20c76d3426/serve-healthcheck-canary/0.log"
Oct 30 19:20:45.296418 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:45.296314    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:45.327281 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:45.327232    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-nxtzp_7948dcd5-eb52-4510-a73d-25b869e1394e/migrator/0.log"
Oct 30 19:20:45.393040 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:45.392999    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dxp6q" event={"ID":"cbf67380-1538-474c-ad65-4f8194a3d0af","Type":"ContainerStarted","Data":"125a270be6fb080c400bcf48c3fcbb324e4568bd92e52e78ab6ae65208f9fc5f"}
Oct 30 19:20:45.393185 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:45.393073    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dxp6q" event={"ID":"cbf67380-1538-474c-ad65-4f8194a3d0af","Type":"ContainerStarted","Data":"0dcf5d06886b771a06a3cdcb7285942b84e1174f01dfad78de15b3691e852d03"}
Oct 30 19:20:45.394922 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:45.394896    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6dnzx" event={"ID":"c1a81d1f-b828-4a0d-a57a-df3d493947d5","Type":"ContainerStarted","Data":"930953cc584d90640ea1beb6f63592a29a8d84ef97ee2041ae555b0c534f0903"}
Oct 30 19:20:45.395062 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:45.395040    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:45.433314 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:45.433261    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6dnzx" podStartSLOduration=35.384144688 podStartE2EDuration="39.433244533s" totalImagesPullingTime="4.049099845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:45.432290933 +0000 UTC m=+39.916947045" watchObservedRunningTime="2026-10-30 19:20:45.433244533 +0000 UTC m=+39.917900642"
Oct 30 19:20:45.490847 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:45.490816    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:45.685798 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:45.685751    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:45.888245 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:45.888218    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:46.095569 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:46.095492    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:46.399782 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:46.399753    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-27xsq" event={"ID":"bbd278f9-76f5-41ed-93d1-435804741834","Type":"ContainerStarted","Data":"3c1696a3ca0eeae9f54026ea743f4db02e367602b6a528f52a9a1f7d86c3f84b"}
Oct 30 19:20:46.402138 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:46.402109    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bm87v" event={"ID":"1b368ff6-93f5-472f-97ae-3f9ffee9102d","Type":"ContainerStarted","Data":"6480e3a00502a341afb97164b824d33d8e8e55974aff9459012c3d8aef61ebf7"}
Oct 30 19:20:46.409075 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:46.406289    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dxp6q" event={"ID":"cbf67380-1538-474c-ad65-4f8194a3d0af","Type":"ContainerStarted","Data":"96dbe2d668be1c5aaa98f2676af8c80fe809a5f727c6bb5a56aeecf4eedbb6d9"}
Oct 30 19:20:46.438251 ip-10-0-140-248 kubenswrapper[2582]: I1030 19:20:46.438159    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-27xsq" podStartSLOduration=34.616740969 podStartE2EDuration="40.438144035s" totalImagesPullingTime="5.821403066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:46.437091162 +0000 UTC m=+40.921747292" watchObservedRunningTime="2026-10-30 19:20:46.438144035 +0000 UTC m=+40.922800147"
Oct 30 19:20:46.640417 ip-10-0-140-248 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 19:20:46.649728 ip-10-0-140-248 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 19:20:46.649896 ip-10-0-140-248 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 19:20:46.649950 ip-10-0-140-248 systemd[1]: kubelet.service: Consumed 2.431s CPU time.
Oct 30 19:20:46.658882 ip-10-0-140-248 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:20:46.758408 ip-10-0-140-248 kubenswrapper[4712]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:20:46.758408 ip-10-0-140-248 kubenswrapper[4712]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:20:46.758408 ip-10-0-140-248 kubenswrapper[4712]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:20:46.758916 ip-10-0-140-248 kubenswrapper[4712]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:20:46.760968 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.760895    4712 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:20:46.760968 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.760967    4712 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.760970    4712 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.760973    4712 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.760976    4712 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.760979    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.760983    4712 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.760987    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.760991    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.760995    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.760998    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761000    4712 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761003    4712 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761006    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761009    4712 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761011    4712 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761014    4712 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761017    4712 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761019    4712 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761023    4712 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:20:46.761033 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761027    4712 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761036    4712 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761039    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761042    4712 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761045    4712 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761048    4712 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761051    4712 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761053    4712 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761056    4712 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761058    4712 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761061    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761063    4712 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761066    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761069    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761072    4712 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761074    4712 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761077    4712 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761079    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761082    4712 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761084    4712 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:20:46.761504 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761087    4712 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761089    4712 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761092    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761094    4712 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761097    4712 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761099    4712 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761102    4712 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761105    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761108    4712 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761111    4712 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761114    4712 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761116    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761119    4712 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761122    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761125    4712 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761127    4712 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761130    4712 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761133    4712 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761136    4712 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:20:46.761993 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761140    4712 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761144    4712 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761148    4712 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761152    4712 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761156    4712 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761160    4712 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761174    4712 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761179    4712 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761182    4712 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761186    4712 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761188    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761191    4712 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761193    4712 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761196    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761199    4712 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761202    4712 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761204    4712 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761207    4712 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761210    4712 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:20:46.762459 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761212    4712 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761214    4712 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761217    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761220    4712 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761222    4712 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761225    4712 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761227    4712 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761231    4712 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761233    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761236    4712 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761239    4712 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761241    4712 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761245    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761247    4712 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761250    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761252    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761255    4712 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761257    4712 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761260    4712 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761262    4712 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:20:46.762909 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761265    4712 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761268    4712 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761271    4712 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761274    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761276    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761279    4712 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761282    4712 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.761284    4712 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762118    4712 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762126    4712 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762130    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762146    4712 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762149    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762152    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762155    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762157    4712 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762161    4712 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762164    4712 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762167    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:20:46.763399 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762170    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762172    4712 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762175    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762177    4712 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762180    4712 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762182    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762185    4712 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762188    4712 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762190    4712 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762193    4712 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762195    4712 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762198    4712 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762200    4712 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762202    4712 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762211    4712 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762214    4712 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762217    4712 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762219    4712 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762222    4712 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:20:46.763850 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762225    4712 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762228    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762230    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762233    4712 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762236    4712 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762239    4712 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762241    4712 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762243    4712 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762246    4712 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762249    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762251    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762253    4712 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762256    4712 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762259    4712 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762261    4712 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762264    4712 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762266    4712 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762269    4712 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762273    4712 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:20:46.764308 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762277    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762279    4712 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762282    4712 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762284    4712 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762287    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762289    4712 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762292    4712 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762295    4712 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762297    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762305    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762309    4712 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762312    4712 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762314    4712 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762317    4712 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762320    4712 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762323    4712 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762325    4712 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762328    4712 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762330    4712 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762334    4712 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:20:46.764892 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762336    4712 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762339    4712 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762341    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762344    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762346    4712 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762349    4712 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762351    4712 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762354    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762357    4712 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762374    4712 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762377    4712 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762379    4712 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762382    4712 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762385    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762388    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762392    4712 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762396    4712 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762401    4712 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762405    4712 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762408    4712 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:20:46.765407 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762412    4712 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762415    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762430    4712 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762434    4712 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762437    4712 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762440    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762443    4712 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762446    4712 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762448    4712 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762451    4712 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762454    4712 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762456    4712 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762460    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762462    4712 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762471    4712 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762475    4712 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.762478    4712 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763044    4712 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763050    4712 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:20:46.765905 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763053    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763056    4712 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763058    4712 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763061    4712 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763064    4712 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763066    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763069    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763072    4712 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763074    4712 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763077    4712 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763079    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763082    4712 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763084    4712 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763087    4712 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763089    4712 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763101    4712 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763105    4712 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763108    4712 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763111    4712 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:20:46.766444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763114    4712 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763117    4712 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763119    4712 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763122    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763124    4712 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763127    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763129    4712 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763132    4712 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763134    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763137    4712 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763140    4712 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763142    4712 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763150    4712 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763153    4712 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763155    4712 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763158    4712 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763160    4712 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763163    4712 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763165    4712 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:20:46.766906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763168    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763170    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763173    4712 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763178    4712 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763181    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763183    4712 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763186    4712 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763189    4712 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763191    4712 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763194    4712 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763202    4712 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763205    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763207    4712 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763210    4712 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763213    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763215    4712 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763218    4712 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763221    4712 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763223    4712 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763226    4712 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:20:46.767414 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763228    4712 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763231    4712 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763234    4712 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763236    4712 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763240    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763242    4712 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763245    4712 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763247    4712 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763250    4712 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763253    4712 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763255    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763258    4712 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763260    4712 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763263    4712 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763265    4712 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763269    4712 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763272    4712 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763274    4712 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763277    4712 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763279    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:20:46.768148 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763282    4712 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763284    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763287    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763296    4712 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763299    4712 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763302    4712 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763304    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763307    4712 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763310    4712 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763312    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763315    4712 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763318    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763320    4712 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763324    4712 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763328    4712 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763332    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763335    4712 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763338    4712 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763340    4712 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:20:46.768822 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763343    4712 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763346    4712 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763349    4712 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763352    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763354    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763357    4712 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.763373    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763470    4712 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763490    4712 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763502    4712 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763507    4712 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763511    4712 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763514    4712 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763518    4712 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763522    4712 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763525    4712 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763530    4712 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763539    4712 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763543    4712 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763546    4712 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763549    4712 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763552    4712 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763572    4712 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:20:46.769261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763576    4712 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763580    4712 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763586    4712 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763589    4712 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763593    4712 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763596    4712 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763599    4712 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763603    4712 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763606    4712 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763609    4712 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763612    4712 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763615    4712 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763618    4712 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763621    4712 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763624    4712 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763628    4712 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763632    4712 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763635    4712 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763638    4712 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763641    4712 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763644    4712 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763647    4712 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763654    4712 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763658    4712 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:20:46.769847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763660    4712 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763665    4712 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763668    4712 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763674    4712 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763677    4712 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763680    4712 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763683    4712 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763686    4712 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763689    4712 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763692    4712 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763695    4712 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763698    4712 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763701    4712 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763703    4712 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763707    4712 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763710    4712 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763714    4712 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763717    4712 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763720    4712 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763723    4712 flags.go:64] FLAG: --help="false"
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763726    4712 flags.go:64] FLAG: --hostname-override="ip-10-0-140-248.ec2.internal"
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763729    4712 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763732    4712 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:20:46.770513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763735    4712 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763743    4712 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763746    4712 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763750    4712 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763752    4712 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763755    4712 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763758    4712 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763761    4712 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763764    4712 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763767    4712 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763771    4712 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763774    4712 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763777    4712 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763780    4712 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763783    4712 flags.go:64] FLAG: --lock-file=""
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763786    4712 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763789    4712 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763792    4712 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763798    4712 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763801    4712 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763804    4712 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763807    4712 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763810    4712 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763813    4712 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:20:46.771060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763816    4712 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763819    4712 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763823    4712 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763826    4712 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763834    4712 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763837    4712 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763840    4712 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763843    4712 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763846    4712 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763848    4712 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763851    4712 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763854    4712 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763861    4712 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763865    4712 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763868    4712 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763871    4712 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763873    4712 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763876    4712 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763881    4712 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763884    4712 flags.go:64] FLAG: --port="10250"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763888    4712 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763891    4712 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-057cf47e1ce072bcf"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763895    4712 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763898    4712 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763901    4712 flags.go:64] FLAG: --register-node="true"
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763904    4712 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:20:46.771639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763907    4712 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763910    4712 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763913    4712 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763916    4712 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763920    4712 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763923    4712 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763926    4712 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763929    4712 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763932    4712 flags.go:64] FLAG: --runonce="false"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763935    4712 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763938    4712 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763941    4712 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763944    4712 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763947    4712 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763951    4712 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763954    4712 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763957    4712 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763960    4712 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763963    4712 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763966    4712 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763969    4712 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763972    4712 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763975    4712 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763978    4712 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763984    4712 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:20:46.772328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763987    4712 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763991    4712 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.763998    4712 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.764000    4712 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.764004    4712 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.764007    4712 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.764010    4712 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.764013    4712 flags.go:64] FLAG: --v="2"
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.764017    4712 flags.go:64] FLAG: --version="false"
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.764022    4712 flags.go:64] FLAG: --vmodule=""
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.764025    4712 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.764029    4712 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764180    4712 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764184    4712 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764187    4712 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764190    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764192    4712 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764195    4712 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764198    4712 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764200    4712 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764203    4712 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764206    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:20:46.773024 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764208    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764211    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764213    4712 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764216    4712 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764218    4712 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764221    4712 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764224    4712 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764227    4712 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764229    4712 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764233    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764236    4712 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764240    4712 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764243    4712 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764245    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764248    4712 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764251    4712 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764254    4712 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764257    4712 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764259    4712 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764262    4712 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:20:46.773581 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764277    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764281    4712 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764285    4712 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764288    4712 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764290    4712 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764293    4712 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764296    4712 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764299    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764302    4712 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764304    4712 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764307    4712 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764310    4712 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764312    4712 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764315    4712 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764318    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764321    4712 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764324    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764326    4712 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764329    4712 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764332    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:20:46.774222 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764334    4712 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764338    4712 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764340    4712 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764345    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764348    4712 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764351    4712 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764353    4712 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764356    4712 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764371    4712 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764374    4712 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764377    4712 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764380    4712 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764383    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764385    4712 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764388    4712 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764391    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764393    4712 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764396    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764399    4712 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764401    4712 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:20:46.774753 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764404    4712 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764408    4712 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764412    4712 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764415    4712 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764420    4712 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764423    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764426    4712 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764429    4712 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764432    4712 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764434    4712 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764437    4712 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764440    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764443    4712 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764445    4712 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764448    4712 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764452    4712 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764455    4712 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764458    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764461    4712 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:20:46.775373 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764463    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:20:46.775943 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764466    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:20:46.775943 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764469    4712 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:20:46.775943 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764471    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:20:46.775943 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764474    4712 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:20:46.775943 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764477    4712 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:20:46.775943 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764480    4712 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:20:46.775943 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764482    4712 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:20:46.775943 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764485    4712 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:20:46.775943 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764487    4712 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:20:46.775943 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764490    4712 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:20:46.775943 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764493    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:20:46.775943 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764495    4712 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:20:46.775943 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764498    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:20:46.775943 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764501    4712 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:20:46.775943 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764504    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:20:46.775943 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764506    4712 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:20:46.776389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.764536    4712 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:20:46.776389 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764572    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:20:46.776389 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764576    4712 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:20:46.776389 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764580    4712 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:20:46.776389 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764583    4712 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:20:46.776389 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764586    4712 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:20:46.776389 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764589    4712 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:20:46.776389 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764591    4712 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:20:46.776389 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764594    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:20:46.776389 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764597    4712 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:20:46.776389 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764600    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:20:46.776389 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764602    4712 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:20:46.776389 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764606    4712 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:20:46.776389 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764609    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:20:46.776389 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764611    4712 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:20:46.776389 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764614    4712 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764617    4712 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764619    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764622    4712 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764625    4712 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764628    4712 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764630    4712 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764633    4712 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764635    4712 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764638    4712 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764641    4712 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764643    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764646    4712 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764649    4712 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764651    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764654    4712 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764656    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764659    4712 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764662    4712 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:20:46.776830 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764664    4712 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764669    4712 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764673    4712 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764677    4712 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764681    4712 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764684    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764687    4712 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764690    4712 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764693    4712 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764696    4712 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764700    4712 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764702    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764705    4712 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764708    4712 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764710    4712 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764713    4712 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764716    4712 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764719    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764722    4712 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:20:46.777477 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764724    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764727    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764729    4712 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764732    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764735    4712 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764737    4712 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764740    4712 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764754    4712 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764759    4712 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764762    4712 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764768    4712 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764771    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764774    4712 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764777    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764780    4712 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764782    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764785    4712 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764787    4712 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764790    4712 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:20:46.777992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764793    4712 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764795    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764798    4712 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764800    4712 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764804    4712 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764807    4712 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764809    4712 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764812    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764815    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764817    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764820    4712 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764825    4712 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764828    4712 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764831    4712 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764834    4712 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764836    4712 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764839    4712 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764841    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764844    4712 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:20:46.778493 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764847    4712 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764849    4712 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764852    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764854    4712 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764858    4712 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764861    4712 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764863    4712 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764866    4712 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764869    4712 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764872    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764874    4712 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764877    4712 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764880    4712 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764882    4712 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764885    4712 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.764888    4712 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.769530    4712 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.769547    4712 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.769566    4712 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.769571    4712 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:20:46.778970 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769632    4712 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769637    4712 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769641    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769644    4712 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769648    4712 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769650    4712 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769653    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769656    4712 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769659    4712 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769661    4712 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769664    4712 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769666    4712 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769669    4712 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769671    4712 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769674    4712 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769676    4712 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769680    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769682    4712 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769685    4712 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769688    4712 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:20:46.779519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769690    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769693    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769696    4712 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769699    4712 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769702    4712 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769705    4712 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769707    4712 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769710    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769718    4712 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769722    4712 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769724    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769727    4712 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769730    4712 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769733    4712 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769735    4712 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769738    4712 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769740    4712 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769743    4712 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769746    4712 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769749    4712 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:20:46.780018 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769751    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769754    4712 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769757    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769759    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769762    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769764    4712 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769767    4712 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769770    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769773    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769775    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769778    4712 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769781    4712 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769783    4712 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769786    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769788    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769791    4712 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769794    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769798    4712 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769802    4712 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:20:46.780519 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769805    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769808    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769812    4712 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769815    4712 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769817    4712 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769820    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769823    4712 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769841    4712 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769844    4712 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769848    4712 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769852    4712 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769855    4712 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769858    4712 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769861    4712 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769863    4712 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769866    4712 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769869    4712 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769871    4712 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769874    4712 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769876    4712 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:20:46.780960 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769880    4712 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769882    4712 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769885    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769888    4712 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769890    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769893    4712 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769895    4712 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769898    4712 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769900    4712 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769903    4712 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769906    4712 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769908    4712 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769911    4712 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769915    4712 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769919    4712 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769922    4712 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769924    4712 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769927    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769930    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:20:46.781480 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769932    4712 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:20:46.781940 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769935    4712 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:20:46.781940 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769937    4712 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:20:46.781940 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769940    4712 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:20:46.781940 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769943    4712 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:20:46.781940 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769945    4712 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:20:46.781940 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769948    4712 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:20:46.781940 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.769950    4712 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:20:46.781940 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.769977    4712 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:20:46.781940 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770115    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:20:46.781940 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770119    4712 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:20:46.781940 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770123    4712 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:20:46.781940 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770126    4712 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:20:46.781940 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770129    4712 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:20:46.781940 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770133    4712 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:20:46.781940 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770135    4712 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:20:46.781940 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770138    4712 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770141    4712 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770143    4712 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770146    4712 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770148    4712 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770151    4712 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770154    4712 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770157    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770159    4712 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770162    4712 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770165    4712 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770167    4712 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770170    4712 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770173    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770176    4712 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770179    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770181    4712 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770184    4712 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770186    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770189    4712 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:20:46.782314 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770192    4712 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770194    4712 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770197    4712 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770199    4712 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770202    4712 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770204    4712 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770207    4712 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770210    4712 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770212    4712 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770215    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770217    4712 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770220    4712 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770223    4712 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770225    4712 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770228    4712 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770231    4712 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770233    4712 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770237    4712 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770241    4712 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:20:46.782809 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770244    4712 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770248    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770251    4712 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770253    4712 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770256    4712 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770259    4712 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770261    4712 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770264    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770267    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770270    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770272    4712 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770275    4712 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770277    4712 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770280    4712 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770282    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770285    4712 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770287    4712 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770290    4712 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770292    4712 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770295    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:20:46.783270 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770298    4712 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770300    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770303    4712 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770305    4712 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770307    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770310    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770313    4712 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770316    4712 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770318    4712 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770321    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770323    4712 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770326    4712 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770328    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770331    4712 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770333    4712 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770336    4712 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770338    4712 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770341    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770343    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:20:46.783758 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770346    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770348    4712 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770351    4712 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770353    4712 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770356    4712 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770373    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770376    4712 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770379    4712 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770382    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770384    4712 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770387    4712 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770389    4712 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770392    4712 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770394    4712 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770397    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770400    4712 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770402    4712 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770405    4712 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770407    4712 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770410    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:20:46.784208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770413    4712 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:20:46.784692 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.770436    4712 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:20:46.784692 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770467    4712 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:20:46.784692 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770471    4712 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:20:46.784692 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770474    4712 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:20:46.784692 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770477    4712 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:20:46.784692 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770480    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:20:46.784692 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770482    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:20:46.784692 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770485    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:20:46.784692 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770488    4712 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:20:46.784692 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770490    4712 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:20:46.784692 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770493    4712 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:20:46.784692 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770496    4712 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:20:46.784692 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770499    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:20:46.784692 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770501    4712 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:20:46.784692 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770504    4712 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:20:46.784692 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770507    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770509    4712 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770512    4712 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770515    4712 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770517    4712 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770520    4712 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770523    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770528    4712 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770530    4712 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770533    4712 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770536    4712 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770538    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770541    4712 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770543    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770546    4712 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770548    4712 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770551    4712 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770557    4712 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770560    4712 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770562    4712 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:20:46.785085 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770565    4712 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770568    4712 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770570    4712 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770573    4712 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770575    4712 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770578    4712 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770581    4712 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770584    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770586    4712 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770589    4712 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770591    4712 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770594    4712 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770597    4712 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770599    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770602    4712 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770605    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770607    4712 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770610    4712 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770613    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770617    4712 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:20:46.785961 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770620    4712 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770623    4712 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770625    4712 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770628    4712 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770630    4712 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770633    4712 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770635    4712 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770638    4712 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770640    4712 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770645    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770648    4712 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770650    4712 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770653    4712 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770655    4712 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770658    4712 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770661    4712 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770664    4712 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770667    4712 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770672    4712 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:20:46.786674 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770676    4712 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770679    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770682    4712 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770685    4712 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770688    4712 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770691    4712 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770694    4712 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770697    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770700    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770702    4712 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770705    4712 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770708    4712 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770712    4712 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770715    4712 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770717    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770720    4712 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770722    4712 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770725    4712 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770728    4712 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770730    4712 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:20:46.787182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770733    4712 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:20:46.787731 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770735    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:20:46.787731 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770739    4712 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:20:46.787731 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770742    4712 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:20:46.787731 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770745    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:20:46.787731 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770748    4712 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:20:46.787731 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770750    4712 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:20:46.787731 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770753    4712 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:20:46.787731 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770755    4712 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:20:46.787731 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770759    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:20:46.787731 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770761    4712 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:20:46.787731 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770764    4712 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:20:46.787731 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:46.770767    4712 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:20:46.787731 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.770833    4712 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:20:46.787731 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.771759    4712 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 19:20:46.787731 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.771811    4712 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 19:20:46.787731 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.772006    4712 server.go:1014] "Starting client certificate rotation"
Oct 30 19:20:46.787731 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.772158    4712 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:20:46.788128 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.772228    4712 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:15:05 +0000 UTC" deadline="2028-03-25 06:41:46.758153889 +0000 UTC"
Oct 30 19:20:46.788128 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.772280    4712 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12275h20m59.985877209s"
Oct 30 19:20:46.788128 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.772514    4712 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:20:46.788128 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.773676    4712 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:20:46.788128 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.775144    4712 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:20:46.788128 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.776735    4712 log.go:25] "Validated CRI v1 image API"
Oct 30 19:20:46.788128 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.777472    4712 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:20:46.788128 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.779057    4712 fs.go:135] Filesystem UUIDs: map[1dfc8d5e-3db9-4c70-9e2f-1ed6139f2942:/dev/nvme0n1p3 23ef4387-7852-4a0c-9768-67d330c12eb0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 19:20:46.788341 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.779076    4712 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0008bcf05fe731a028bb8218b6d25cc4481c6f5e08353eed82e64ec4cf016693/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0008bcf05fe731a028bb8218b6d25cc4481c6f5e08353eed82e64ec4cf016693/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0889efca5dbac94607f60d7ed64589805be039dcfd58f4d2d47c84eefa116e07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0889efca5dbac94607f60d7ed64589805be039dcfd58f4d2d47c84eefa116e07/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0dcf5d06886b771a06a3cdcb7285942b84e1174f01dfad78de15b3691e852d03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0dcf5d06886b771a06a3cdcb7285942b84e1174f01dfad78de15b3691e852d03/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25a1ca4031e90d820f747872f6f21e62c65560f9ac1f5bec98d59483f52873d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25a1ca4031e90d820f747872f6f21e62c65560f9ac1f5bec98d59483f52873d9/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41f07742d96fc257c2ba7769b5c4178e2a1b53570178cb198523caa4f19dbbc6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41f07742d96fc257c2ba7769b5c4178e2a1b53570178cb198523caa4f19dbbc6/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/44a177bde116a168d2eef7e8125c71aa8e9cff40c9c22201148e20d92b106799/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/44a177bde116a168d2eef7e8125c71aa8e9cff40c9c22201148e20d92b106799/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58ef30be00e42d2ac86281d2a108867103e940ebaec4fcf815e1bd92d333c82e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58ef30be00e42d2ac86281d2a108867103e940ebaec4fcf815e1bd92d333c82e/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a2d80d43f8e032c61835cfefd5452762660b8d3b2b040861342d06f421cdf04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a2d80d43f8e032c61835cfefd5452762660b8d3b2b040861342d06f421cdf04/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/731a90598c958a1bf0e7c98cef4b5f80c910a4e391bd5501f60dd005931f49ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/731a90598c958a1bf0e7c98cef4b5f80c910a4e391bd5501f60dd005931f49ad/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87f8f4d5303e13435b5f3db595fc92e5b67ca5791dab7b732ed6ecb60cd55374/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87f8f4d5303e13435b5f3db595fc92e5b67ca5791dab7b732ed6ecb60cd55374/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bc6f2de22212ef491069993807ee1b09446c52f5dee4c880b8e8f28a5c4cb40/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bc6f2de22212ef491069993807ee1b09446c52f5dee4c880b8e8f28a5c4cb40/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f71ec68a3bf3f19f744fb775f8ec2b1f2b8ac746ea745d970c96f5057a3c01d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f71ec68a3bf3f19f744fb775f8ec2b1f2b8ac746ea745d970c96f5057a3c01d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1fdf1c7824e8af0757de524fb1b5161ca4c06bd4777aadccf3d7c78419cbd60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1fdf1c7824e8af0757de524fb1b5161ca4c06bd4777aadccf3d7c78419cbd60/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1e95132776f9a23e7fc3f5da5ac31b3e51af3c53f27b1574ff98a9d877f3db3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1e95132776f9a23e7fc3f5da5ac31b3e51af3c53f27b1574ff98a9d877f3db3/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c727fc80a4e6ec4e17ef4cd84e37e14f8d6ba36d4230b5d05e4932ae8e33b3dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c727fc80a4e6ec4e17ef4cd84e37e14f8d6ba36d4230b5d05e4932ae8e33b3dc/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce51e3b3e7c2408cc012df9a5512716cfba5ea80bc2f9aafd8f72ad0130a320f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce51e3b3e7c2408cc012df9a5512716cfba5ea80bc2f9aafd8f72ad0130a320f/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d60f6c831ff3e931fe18c1fc7736ba7c15f83287f202a5f954397eaa96a3c8b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d60f6c831ff3e931fe18c1fc7736ba7c15f83287f202a5f954397eaa96a3c8b9/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd650ca092f9d13b2e26a0ac96286f7cc347d013f8951d5e184bb33cbb2d503b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd650ca092f9d13b2e26a0ac96286f7cc347d013f8951d5e184bb33cbb2d503b/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/029fdc09-aaca-4dc2-90ff-86e05c26bf9e/volumes/kubernetes.io~projected/kube-api-access-5kpwc:{mountpoint:/var/lib/kubelet/pods/029fdc09-aaca-4dc2-90ff-86e05c26bf9e/volumes/kubernetes.io~projected/kube-api-access-5kpwc major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/093cd549-ebec-45f5-b888-42019476c70c/volumes/kubernetes.io~projected/kube-api-access-tkgsr:{mountpoint:/var/lib/kubelet/pods/093cd549-ebec-45f5-b888-42019476c70c/volumes/kubernetes.io~projected/kube-api-access-tkgsr major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b45e511-0890-4a70-96d7-01ba249d65cb/volumes/kubernetes.io~projected/kube-api-access-lqxbc:{mountpoint:/var/lib/kubelet/pods/0b45e511-0890-4a70-96d7-01ba249d65cb/volumes/kubernetes.io~projected/kube-api-access-lqxbc major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cffcb83-c327-45ab-90b7-837f3f455f92/volumes/kubernetes.io~projected/kube-api-access-5ll8f:{mountpoint:/var/lib/kubelet/pods/0cffcb83-c327-45ab-90b7-837f3f455f92/volumes/kubernetes.io~projected/kube-api-access-5ll8f major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cffcb83-c327-45ab-90b7-837f3f455f92/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/0cffcb83-c327-45ab-90b7-837f3f455f92/volumes/kubernetes.io~secret/signing-key major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b368ff6-93f5-472f-97ae-3f9ffee9102d/volumes/kubernetes.io~projected/kube-api-access-nwzm8:{mountpoint:/var/lib/kubelet/pods/1b368ff6-93f5-472f-97ae-3f9ffee9102d/volumes/kubernetes.io~projected/kube-api-access-nwzm8 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b368ff6-93f5-472f-97ae-3f9ffee9102d/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/1b368ff6-93f5-472f-97ae-3f9ffee9102d/volumes/kubernetes.io~secret/metrics-tls major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7948dcd5-eb52-4510-a73d-25b869e1394e/volumes/kubernetes.io~projected/kube-api-access-d98ps:{mountpoint:/var/lib/kubelet/pods/7948dcd5-eb52-4510-a73d-25b869e1394e/volumes/kubernetes.io~projected/kube-api-access-d98ps major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c299f86-7f6e-4666-ab9c-542537e6ff8e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/8c299f86-7f6e-4666-ab9c-542537e6ff8e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c299f86-7f6e-4666-ab9c-542537e6ff8e/volumes/kubernetes.io~projected/kube-api-access-d5xl5:{mountpoint:/var/lib/kubelet/pods/8c299f86-7f6e-4666-ab9c-542537e6ff8e/volumes/kubernetes.io~projected/kube-api-access-d5xl5 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c299f86-7f6e-4666-ab9c-542537e6ff8e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/8c299f86-7f6e-4666-ab9c-542537e6ff8e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90c7079a-e03a-4d3e-bb41-b61ce36292ec/volumes/kubernetes.io~projected/kube-api-access-pv6g9:{mountpoint:/var/lib/kubelet/pods/90c7079a-e03a-4d3e-bb41-b61ce36292ec/volumes/kubernetes.io~projected/kube-api-access-pv6g9 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/954bf0fe-4287-4435-9abb-fd06dc478099/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/954bf0fe-4287-4435-9abb-fd06dc478099/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9838f46c-794d-4b6f-9f2d-a1a45e21819f/volumes/kubernetes.io~projected/kube-api-access-kdmgh:{mountpoint:/var/lib/kubelet/pods/9838f46c-794d-4b6f-9f2d-a1a45e21819f/volumes/kubernetes.io~projected/kube-api-access-kdmgh major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada/volumes/kubernetes.io~projected/kube-api-access-4bf6h:{mountpoint:/var/lib/kubelet/pods/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada/volumes/kubernetes.io~projected/kube-api-access-4bf6h major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aacc7bb9-0109-4984-884f-50f8d2144559/volumes/kubernetes.io~projected/kube-api-access-drlzm:{mountpoint:/var/lib/kubelet/pods/aacc7bb9-0109-4984-884f-50f8d2144559/volumes/kubernetes.io~projected/kube-api-access-drlzm major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbd278f9-76f5-41ed-93d1-435804741834/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/bbd278f9-76f5-41ed-93d1-435804741834/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1a81d1f-b828-4a0d-a57a-df3d493947d5/volumes/kubernetes.io~projected/kube-api-access-r9zdg:{mountpoint:/var/lib/kubelet/pods/c1a81d1f-b828-4a0d-a57a-df3d493947d5/volumes/kubernetes.io~projected/kube-api-access-r9zdg major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4aa3418-bbc0-4c70-a42a-2a65e478b47d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c4aa3418-bbc0-4c70-a42a-2a65e478b47d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4aa3418-bbc0-4c70-a42a-2a65e478b47d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c4aa3418-bbc0-4c70-a42a-2a65e478b47d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4aa3418-bbc0-4c70-a42a-2a65e478b47d/volumes/kubernetes.io~projected/kube-api-access-gwxjt:{mountpoint:/var/lib/kubelet/pods/c4aa3418-bbc0-4c70-a42a-2a65e478b47d/volumes/kubernetes.io~projected/kube-api-access-gwxjt major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbf67380-1538-474c-ad65-4f8194a3d0af/volumes/kubernetes.io~projected/kube-api-access-s9txd:{mountpoint:/var/lib/kubelet/pods/cbf67380-1538-474c-ad65-4f8194a3d0af/volumes/kubernetes.io~projected/kube-api-access-s9txd major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbf67380-1538-474c-ad65-4f8194a3d0af/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/cbf67380-1538-474c-ad65-4f8194a3d0af/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8cf3812-0483-481a-86e6-8c20c76d3426/volumes/kubernetes.io~projected/kube-api-access-hk84z:{mountpoint:/var/lib/kubelet/pods/e8cf3812-0483-481a-86e6-8c20c76d3426/volumes/kubernetes.io~projected/kube-api-access-hk84z major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8cf3812-0483-481a-86e6-8c20c76d3426/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e8cf3812-0483-481a-86e6-8c20c76d3426/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f29fd70f-b10e-4154-8652-6b17fbe3cf13/volumes/kubernetes.io~projected/kube-api-access-spzw5:{mountpoint:/var/lib/kubelet/pods/f29fd70f-b10e-4154-8652-6b17fbe3cf13/volumes/kubernetes.io~projected/kube-api-access-spzw5 major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/adec934b09a6268d0d645e1284d01dc6deb85e0f9d29a0fad19bf177c634cd04/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/25bbbb74de445f9b6822ce1011f894393a0c2017e1f55833fef238d3e312d11a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f86d93cbd18a2d9872945f2bc493db5cd884b7a419284f792d3b89dfa3a7c385/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/a15b587303a22443c6104c5a6f1745cabacb154127276016aeb7ad6b1f3ea9a8/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/60edcbdfcd0e14654c93a4c1dd3accd9e4f82cd83e3c0a11eddffc5e6590d34d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/dd426a5e7f075eb706d1a81ee817c57ed88f75a6a8923349d33c1e3b7c917e4a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/0ab8a23d102c8b213af139a7f178fd3db2d1b60ea28a1d773c108a91affbe38d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/afda63b609f75b710102db55ea28177e2a1cbe71d54574f55a9099f39973fb41/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/920a16f1e241861420519a7743618c01e3b28715d94a5e6e39549f98bf16ba8b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/886508c1214086fbc60ba45b35a79a081e9db7065a5beca4291c07698bac6b1f/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/ca0205b4f2009438c775ba5a8d2a37932ed6f0cb2b888de24a26a360d57ea7c8/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/43341a96b5bd5ed02c2f53636dd18682b340ed1884c6bb33a175c2667af1995a/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-151:{mountpoint:/var/lib/containers/storage/overlay/96a5a4b328259c70aeeb0170a98d259f7a22b9793c72b071fb828178cd52d465/merged major:0 minor:151 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9322220375a0d35153acd2c4f44a2c9e8a425b0c2e0d39df52655fecdfeec019/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ab48e20d49fe49cc444f5352b4a4427cf39b0dd560d803858e2040fe4a4e3fd8/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/ffc83245caec38e7b0f53caa9ebc89043937ada19053d68deae140c9774bb298/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a8440c4c76fcd00365be1706b5a86c86029787f107bf8ae20f52bd26f0e800ed/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0779d70259fcd06672e086bde67516e793be13159aff5b19d742a325a78d09d2/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/6450a990d6477ff84db6839d087796e598c07214ba9f3430060eb396569ba7d6/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/95a61c52c8d23bec6214b7aeb022e7f284d668cdb70066b0313ff1644ac9a52d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f99a4b541bbfb6737f52eca61f4ca9b00a7d64bad41a00864494752bfa1c50dc/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/f42c114a4d191bb93b8e8780cf2df95f688abec4a6ac32ea7e975e6311f19875/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/85bc623d0af3038ab02953aafee5feebdf7634997da3da21b84ec42524ea6384/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/9111a9e1ee8982b93d0f44fc9515f813a34af62169a5914d0955ed9b48264864/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/a3fd985d1d724c121d4fa31e37fb5fbcba2e940c8888607b80e22555cf7775ee/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/e009653ea0b5ed9cd75057ea8df795f1692d297cc4cac63b49e5271b45ca24ce/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/c7ae34b51c81df0b1d81d0b2b2f364fd7a9a71b13deb314188ddebf8f46bba5f/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/0e36f56d0f3e63bb3e84bc5f69ba673928264f2f64cb6a8e40fba31ae88f68df/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/7eb6a4f69b4602c929dc83a7d76a10bb74e406898d0c4ef290d59403839e14e1/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/b0ee8ac13218bab29930cd4a7ae618a22283705a5ff5ef1ffac6201754569340/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/04be9da0d8f79cf9e7547f12b686e183c780fe1f2864e568952aaa606b5ed2ad/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/744d817c2c0f0875a6c2eb672ee63716ce756a1f3f5b59eed6a00170950011e5/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/26a9fec5c2b21aae05dc8c1a0468932ce94d81410f32ed4e3fedd91a6c37d4ac/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/ab1dbd0b6be0ae181d543ad68511b1f81ccc97c839d72019d2cf31f176d11fd7/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/638dcd028e06dee88ebe43fad6630732a33cf9a67a0080fbe1bc45ab49380792/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/7684cc0c9466ad3e4b3c5d0e918736855e632b73d2d33f3fcdc66248db484ff9/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/e9860d3375ded99ce7488c15142449eee5a57c0c560c375e63f5bb0b41891653/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/e0a600ac0f37ccc6f17d3db555e0ce3d3a89133ddb489ba512c50bb5fa1320d3/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/9e9679ebbc116fe3ff7fb2030c8dead0fe091494192ec91239e96c57d68a3b0e/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/e47f36b53f627389a53793bdda8d886b405d6f3ce9fc2001b2e75195026d636f/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/9424692af69502c54619c71180f0c1463d0860b673da333dfd9cbf7446fcd43d/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b5c2f9987b66e3d7470adf45dac98dc703810b781d1ddeba1472e47542bbd159/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/654dcdbc85d00018f982c25c29edf12fcb18a67c09be5ccb83cd3c1e78846e4c/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/efd8218df98a87be48ea6e5f46361c32fb51bfeac67a0888b93c778ebcacc326/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/c72d0362e9bda725a07f44ba1497a2d2f6823be74743989fa23afd29255d6931/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/6ba4d532ffb3151c8fd76876134565478f10c3711bdd0ba921573edeee005234/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/a4ec58e568d1f3a54f0f9a23ed38aee836a1b21f3f1d36ef5c84345e4e3c2262/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/604c3634dd5e40a19487088f91cd7c266300d616c3956f44fa21149985340f8a/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/87b94f140708beb6212717344bd01da173a1e89ed9903473ca84268745f519f1/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 19:20:46.788546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.787121    4712 manager.go:217] Machine: {Timestamp:2026-10-30 19:20:46.786425466 +0000 UTC m=+0.085687587 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099818 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21513917c62edb1174617ecd4260ba SystemUUID:ec215139-17c6-2edb-1174-617ecd4260ba BootID:6cc37678-bf2e-461b-8323-6dbbffd96943 Filesystems:[{Device:/run/containers/storage/overlay-containers/ce51e3b3e7c2408cc012df9a5512716cfba5ea80bc2f9aafd8f72ad0130a320f/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7948dcd5-eb52-4510-a73d-25b869e1394e/volumes/kubernetes.io~projected/kube-api-access-d98ps DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bbd278f9-76f5-41ed-93d1-435804741834/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b368ff6-93f5-472f-97ae-3f9ffee9102d/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:245 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41f07742d96fc257c2ba7769b5c4178e2a1b53570178cb198523caa4f19dbbc6/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/954bf0fe-4287-4435-9abb-fd06dc478099/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f71ec68a3bf3f19f744fb775f8ec2b1f2b8ac746ea745d970c96f5057a3c01d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87f8f4d5303e13435b5f3db595fc92e5b67ca5791dab7b732ed6ecb60cd55374/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0cffcb83-c327-45ab-90b7-837f3f455f92/volumes/kubernetes.io~projected/kube-api-access-5ll8f DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/c4aa3418-bbc0-4c70-a42a-2a65e478b47d/volumes/kubernetes.io~projected/kube-api-access-gwxjt DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0b45e511-0890-4a70-96d7-01ba249d65cb/volumes/kubernetes.io~projected/kube-api-access-lqxbc DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0889efca5dbac94607f60d7ed64589805be039dcfd58f4d2d47c84eefa116e07/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b368ff6-93f5-472f-97ae-3f9ffee9102d/volumes/kubernetes.io~projected/kube-api-access-nwzm8 DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8c299f86-7f6e-4666-ab9c-542537e6ff8e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/029fdc09-aaca-4dc2-90ff-86e05c26bf9e/volumes/kubernetes.io~projected/kube-api-access-5kpwc DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9838f46c-794d-4b6f-9f2d-a1a45e21819f/volumes/kubernetes.io~projected/kube-api-access-kdmgh DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0008bcf05fe731a028bb8218b6d25cc4481c6f5e08353eed82e64ec4cf016693/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd650ca092f9d13b2e26a0ac96286f7cc347d013f8951d5e184bb33cbb2d503b/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1e95132776f9a23e7fc3f5da5ac31b3e51af3c53f27b1574ff98a9d877f3db3/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4aa3418-bbc0-4c70-a42a-2a65e478b47d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aacc7bb9-0109-4984-884f-50f8d2144559/volumes/kubernetes.io~projected/kube-api-access-drlzm DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25a1ca4031e90d820f747872f6f21e62c65560f9ac1f5bec98d59483f52873d9/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1a81d1f-b828-4a0d-a57a-df3d493947d5/volumes/kubernetes.io~projected/kube-api-access-r9zdg DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0dcf5d06886b771a06a3cdcb7285942b84e1174f01dfad78de15b3691e852d03/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c299f86-7f6e-4666-ab9c-542537e6ff8e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1fdf1c7824e8af0757de524fb1b5161ca4c06bd4777aadccf3d7c78419cbd60/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/44a177bde116a168d2eef7e8125c71aa8e9cff40c9c22201148e20d92b106799/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f29fd70f-b10e-4154-8652-6b17fbe3cf13/volumes/kubernetes.io~projected/kube-api-access-spzw5 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8c299f86-7f6e-4666-ab9c-542537e6ff8e/volumes/kubernetes.io~projected/kube-api-access-d5xl5 DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90c7079a-e03a-4d3e-bb41-b61ce36292ec/volumes/kubernetes.io~projected/kube-api-access-pv6g9 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bc6f2de22212ef491069993807ee1b09446c52f5dee4c880b8e8f28a5c4cb40/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/093cd549-ebec-45f5-b888-42019476c70c/volumes/kubernetes.io~projected/kube-api-access-tkgsr DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c727fc80a4e6ec4e17ef4cd84e37e14f8d6ba36d4230b5d05e4932ae8e33b3dc/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8cf3812-0483-481a-86e6-8c20c76d3426/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cbf67380-1538-474c-ad65-4f8194a3d0af/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:262 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a2d80d43f8e032c61835cfefd5452762660b8d3b2b040861342d06f421cdf04/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-151 DeviceMajor:0 DeviceMinor:151 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbf67380-1538-474c-ad65-4f8194a3d0af/volumes/kubernetes.io~projected/kube-api-access-s9txd DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0cffcb83-c327-45ab-90b7-837f3f455f92/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d60f6c831ff3e931fe18c1fc7736ba7c15f83287f202a5f954397eaa96a3c8b9/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c4aa3418-bbc0-4c70-a42a-2a65e478b47d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada/volumes/kubernetes.io~projected/kube-api-access-4bf6h DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8cf3812-0483-481a-86e6-8c20c76d3426/volumes/kubernetes.io~projected/kube-api-access-hk84z DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/731a90598c958a1bf0e7c98cef4b5f80c910a4e391bd5501f60dd005931f49ad/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58ef30be00e42d2ac86281d2a108867103e940ebaec4fcf815e1bd92d333c82e/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0dcf5d06886b771 MacAddress:ae:96:46:fa:7c:51 Speed:10000 Mtu:8901} {Name:41f07742d96fc25 MacAddress:46:d4:91:ac:1c:5b Speed:10000 Mtu:8901} {Name:58ef30be00e42d2 MacAddress:12:fc:f7:2b:9b:b7 Speed:10000 Mtu:8901} {Name:731a90598c958a1 MacAddress:fa:a3:c5:b5:5c:3b Speed:10000 Mtu:8901} {Name:b1fdf1c7824e8af MacAddress:06:aa:a9:4a:a6:7b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d6:bd:6d:af Speed:0 Mtu:9001} {Name:br-int MacAddress:52:ba:14:7d:fc:40 Speed:0 Mtu:8901} {Name:c1e95132776f9a2 MacAddress:52:00:84:a7:79:7a Speed:10000 Mtu:8901} {Name:ce51e3b3e7c2408 MacAddress:ce:9b:71:8a:99:0f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d6:bd:6d:af Speed:0 Mtu:9001} {Name:fd650ca092f9d13 MacAddress:66:dd:9e:fe:a6:1e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:16:d3:17:66:bf:b5 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7e:cb:28:0e:ee:f6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:20:46.788546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.787450    4712 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:20:46.788546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.787492    4712 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:20:46.788546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.787754    4712 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:20:46.788546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.787773    4712 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-248.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:20:46.788546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.787887    4712 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:20:46.788546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.787894    4712 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:20:46.788546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.787904    4712 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:20:46.788546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.787922    4712 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:20:46.788546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.788001    4712 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:20:46.788546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.788225    4712 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:20:46.788546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.788333    4712 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:20:46.788546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.788380    4712 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:20:46.788546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.788393    4712 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:20:46.788546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.788404    4712 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:20:46.788546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.788412    4712 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:20:46.788546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.788424    4712 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:20:46.789240 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.788975    4712 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:20:46.789240 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.788985    4712 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:20:46.789431 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.789412    4712 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:20:46.800804 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.800788    4712 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:20:46.800804 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.800801    4712 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:20:46.800935 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.800892    4712 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 19:20:46.801094 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.801082    4712 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:20:46.801130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.801112    4712 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:20:46.801212 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.801203    4712 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:20:46.801264 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.801217    4712 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:20:46.801264 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.801223    4712 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:20:46.801264 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.801230    4712 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:20:46.801264 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.801238    4712 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:20:46.801264 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.801250    4712 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:20:46.801264 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.801263    4712 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:20:46.801545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.801275    4712 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:20:46.801545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.801284    4712 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:20:46.801545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.801293    4712 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:20:46.801545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.801314    4712 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:20:46.801545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.801327    4712 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:20:46.801545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.801357    4712 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:20:46.801545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.801379    4712 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:20:46.804793 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.804780    4712 server.go:1263] "Started kubelet"
Oct 30 19:20:46.804891 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.804850    4712 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:20:46.805345 ip-10-0-140-248 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:20:46.806746 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.805403    4712 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:20:46.806746 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.805987    4712 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:20:46.806746 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.806286    4712 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:20:46.808380 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.807233    4712 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:20:46.810219 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.809206    4712 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:20:46.810219 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.809707    4712 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:20:46.810219 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.809727    4712 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:20:46.810219 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.809730    4712 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:20:46.810219 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.809802    4712 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:15:07 +0000 UTC" deadline="2028-08-13 21:40:31.253338884 +0000 UTC"
Oct 30 19:20:46.810219 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.809830    4712 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15674h19m44.443511917s"
Oct 30 19:20:46.810219 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:20:46.809859    4712 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-248.ec2.internal\" not found"
Oct 30 19:20:46.810219 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.809851    4712 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:20:46.812673 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.812191    4712 factory.go:55] Registering systemd factory
Oct 30 19:20:46.812673 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.812214    4712 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:20:46.812673 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.812649    4712 factory.go:153] Registering CRI-O factory
Oct 30 19:20:46.812673 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.812665    4712 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:20:46.812923 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.812712    4712 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:20:46.812923 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.812729    4712 factory.go:103] Registering Raw factory
Oct 30 19:20:46.812923 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.812746    4712 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.812994    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b45e511-0890-4a70-96d7-01ba249d65cb" volumeName="kubernetes.io/projected/0b45e511-0890-4a70-96d7-01ba249d65cb-kube-api-access-lqxbc" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813032    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b368ff6-93f5-472f-97ae-3f9ffee9102d" volumeName="kubernetes.io/empty-dir/1b368ff6-93f5-472f-97ae-3f9ffee9102d-tmp-dir" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813055    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aacc7bb9-0109-4984-884f-50f8d2144559" volumeName="kubernetes.io/configmap/aacc7bb9-0109-4984-884f-50f8d2144559-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813069    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f29fd70f-b10e-4154-8652-6b17fbe3cf13" volumeName="kubernetes.io/configmap/f29fd70f-b10e-4154-8652-6b17fbe3cf13-iptables-alerter-script" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813080    4712 manager.go:319] Starting recovery of all containers
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813083    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f29fd70f-b10e-4154-8652-6b17fbe3cf13" volumeName="kubernetes.io/projected/f29fd70f-b10e-4154-8652-6b17fbe3cf13-kube-api-access-spzw5" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813102    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cffcb83-c327-45ab-90b7-837f3f455f92" volumeName="kubernetes.io/configmap/0cffcb83-c327-45ab-90b7-837f3f455f92-signing-cabundle" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813121    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cffcb83-c327-45ab-90b7-837f3f455f92" volumeName="kubernetes.io/secret/0cffcb83-c327-45ab-90b7-837f3f455f92-signing-key" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813135    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9838f46c-794d-4b6f-9f2d-a1a45e21819f" volumeName="kubernetes.io/empty-dir/9838f46c-794d-4b6f-9f2d-a1a45e21819f-archives-path" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813149    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4aa3418-bbc0-4c70-a42a-2a65e478b47d" volumeName="kubernetes.io/empty-dir/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-tuned" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813160    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="029fdc09-aaca-4dc2-90ff-86e05c26bf9e" volumeName="kubernetes.io/projected/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-kube-api-access-5kpwc" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813188    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b368ff6-93f5-472f-97ae-3f9ffee9102d" volumeName="kubernetes.io/configmap/1b368ff6-93f5-472f-97ae-3f9ffee9102d-config-volume" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813199    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c299f86-7f6e-4666-ab9c-542537e6ff8e" volumeName="kubernetes.io/configmap/8c299f86-7f6e-4666-ab9c-542537e6ff8e-ovnkube-config" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813210    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c299f86-7f6e-4666-ab9c-542537e6ff8e" volumeName="kubernetes.io/projected/8c299f86-7f6e-4666-ab9c-542537e6ff8e-kube-api-access-d5xl5" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813219    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954bf0fe-4287-4435-9abb-fd06dc478099" volumeName="kubernetes.io/configmap/954bf0fe-4287-4435-9abb-fd06dc478099-konnectivity-ca" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813242    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9838f46c-794d-4b6f-9f2d-a1a45e21819f" volumeName="kubernetes.io/projected/9838f46c-794d-4b6f-9f2d-a1a45e21819f-kube-api-access-kdmgh" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813254    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada" volumeName="kubernetes.io/configmap/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada-serviceca" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813266    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbf67380-1538-474c-ad65-4f8194a3d0af" volumeName="kubernetes.io/secret/cbf67380-1538-474c-ad65-4f8194a3d0af-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813282    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b45e511-0890-4a70-96d7-01ba249d65cb" volumeName="kubernetes.io/configmap/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-daemon-config" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813293    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9838f46c-794d-4b6f-9f2d-a1a45e21819f" volumeName="kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-service-ca-bundle" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813304    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aacc7bb9-0109-4984-884f-50f8d2144559" volumeName="kubernetes.io/configmap/aacc7bb9-0109-4984-884f-50f8d2144559-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813315    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4aa3418-bbc0-4c70-a42a-2a65e478b47d" volumeName="kubernetes.io/empty-dir/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-tmp" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813325    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4aa3418-bbc0-4c70-a42a-2a65e478b47d" volumeName="kubernetes.io/projected/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-kube-api-access-gwxjt" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813337    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b45e511-0890-4a70-96d7-01ba249d65cb" volumeName="kubernetes.io/configmap/0b45e511-0890-4a70-96d7-01ba249d65cb-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813349    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="954bf0fe-4287-4435-9abb-fd06dc478099" volumeName="kubernetes.io/secret/954bf0fe-4287-4435-9abb-fd06dc478099-agent-certs" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813383    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbd278f9-76f5-41ed-93d1-435804741834" volumeName="kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813396    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7948dcd5-eb52-4510-a73d-25b869e1394e" volumeName="kubernetes.io/projected/7948dcd5-eb52-4510-a73d-25b869e1394e-kube-api-access-d98ps" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813412    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c299f86-7f6e-4666-ab9c-542537e6ff8e" volumeName="kubernetes.io/configmap/8c299f86-7f6e-4666-ab9c-542537e6ff8e-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813423    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c299f86-7f6e-4666-ab9c-542537e6ff8e" volumeName="kubernetes.io/secret/8c299f86-7f6e-4666-ab9c-542537e6ff8e-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813438    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9838f46c-794d-4b6f-9f2d-a1a45e21819f" volumeName="kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813451    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada" volumeName="kubernetes.io/projected/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada-kube-api-access-4bf6h" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813461    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aacc7bb9-0109-4984-884f-50f8d2144559" volumeName="kubernetes.io/configmap/aacc7bb9-0109-4984-884f-50f8d2144559-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813473    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aacc7bb9-0109-4984-884f-50f8d2144559" volumeName="kubernetes.io/projected/aacc7bb9-0109-4984-884f-50f8d2144559-kube-api-access-drlzm" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813492    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8cf3812-0483-481a-86e6-8c20c76d3426" volumeName="kubernetes.io/projected/e8cf3812-0483-481a-86e6-8c20c76d3426-kube-api-access-hk84z" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813503    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="093cd549-ebec-45f5-b888-42019476c70c" volumeName="kubernetes.io/projected/093cd549-ebec-45f5-b888-42019476c70c-kube-api-access-tkgsr" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813513    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c7079a-e03a-4d3e-bb41-b61ce36292ec" volumeName="kubernetes.io/empty-dir/90c7079a-e03a-4d3e-bb41-b61ce36292ec-tmp-dir" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813526    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a81d1f-b828-4a0d-a57a-df3d493947d5" volumeName="kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813536    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbf67380-1538-474c-ad65-4f8194a3d0af" volumeName="kubernetes.io/configmap/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813546    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbf67380-1538-474c-ad65-4f8194a3d0af" volumeName="kubernetes.io/empty-dir/cbf67380-1538-474c-ad65-4f8194a3d0af-data-volume" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813558    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbf67380-1538-474c-ad65-4f8194a3d0af" volumeName="kubernetes.io/projected/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-api-access-s9txd" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813569    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cffcb83-c327-45ab-90b7-837f3f455f92" volumeName="kubernetes.io/projected/0cffcb83-c327-45ab-90b7-837f3f455f92-kube-api-access-5ll8f" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813581    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b368ff6-93f5-472f-97ae-3f9ffee9102d" volumeName="kubernetes.io/projected/1b368ff6-93f5-472f-97ae-3f9ffee9102d-kube-api-access-nwzm8" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813601    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b368ff6-93f5-472f-97ae-3f9ffee9102d" volumeName="kubernetes.io/secret/1b368ff6-93f5-472f-97ae-3f9ffee9102d-metrics-tls" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813613    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c299f86-7f6e-4666-ab9c-542537e6ff8e" volumeName="kubernetes.io/configmap/8c299f86-7f6e-4666-ab9c-542537e6ff8e-env-overrides" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813623    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90c7079a-e03a-4d3e-bb41-b61ce36292ec" volumeName="kubernetes.io/projected/90c7079a-e03a-4d3e-bb41-b61ce36292ec-kube-api-access-pv6g9" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813635    4712 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8cf3812-0483-481a-86e6-8c20c76d3426" volumeName="kubernetes.io/secret/e8cf3812-0483-481a-86e6-8c20c76d3426-cert" seLinuxMountContext=""
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813646    4712 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:20:46.814587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.813653    4712 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:20:46.826631 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.826595    4712 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:20:46.837864 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.837845    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:20:46.839078 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.839056    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:20:46.844645 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.844624    4712 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:20:46.844645 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.844643    4712 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:20:46.844778 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.844659    4712 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:20:46.844778 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:20:46.844694    4712 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 19:20:46.853048 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.850753    4712 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:20:46.853048 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.850928    4712 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 19:20:46.853048 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.850982    4712 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:20:46.859722 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.859706    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:20:46.860030 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.860002    4712 generic.go:356] "Generic (PLEG): container finished" podID="8c299f86-7f6e-4666-ab9c-542537e6ff8e" containerID="98329262d3f3ece16ef013d60d3b780cb5a91c7ef6ef9056998c63c47fdef0b0" exitCode=1
Oct 30 19:20:46.863419 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.863395    4712 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:20:46.863972 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.863950    4712 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:20:46.873779 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.873755    4712 generic.go:356] "Generic (PLEG): container finished" podID="2670e93fcc64eb8a9f070c9bd973c8ad" containerID="11a15e06b0d4eb386807e9a1fdf660153c2f718eca5719db9b858af0e675796c" exitCode=0
Oct 30 19:20:46.882788 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.882766    4712 generic.go:356] "Generic (PLEG): container finished" podID="aacc7bb9-0109-4984-884f-50f8d2144559" containerID="ee6ebeef62c630742d85a28264ede1ca43056ddd690995a3ef874e5c2b9a7852" exitCode=0
Oct 30 19:20:46.882788 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.882788    4712 generic.go:356] "Generic (PLEG): container finished" podID="aacc7bb9-0109-4984-884f-50f8d2144559" containerID="602b3ffee49b831edd1d0c8f8983c3f9f8b013766c73806ccfa51aad09b6e8f7" exitCode=0
Oct 30 19:20:46.882909 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.882796    4712 generic.go:356] "Generic (PLEG): container finished" podID="aacc7bb9-0109-4984-884f-50f8d2144559" containerID="276381a3abdd34d4a03e249c861ef6b23c7c296777f471f62a1cbc2915af23e8" exitCode=0
Oct 30 19:20:46.882909 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.882803    4712 generic.go:356] "Generic (PLEG): container finished" podID="aacc7bb9-0109-4984-884f-50f8d2144559" containerID="def8dedaa8407ed4143926c9fe8aa64e334b326e8cce6933728b39c45ebabfd1" exitCode=0
Oct 30 19:20:46.882909 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.882808    4712 generic.go:356] "Generic (PLEG): container finished" podID="aacc7bb9-0109-4984-884f-50f8d2144559" containerID="348a4b52a691d67d14c50f9ceccb82a7bc847707dae1610a7a48a6090541290c" exitCode=0
Oct 30 19:20:46.882909 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.882815    4712 generic.go:356] "Generic (PLEG): container finished" podID="aacc7bb9-0109-4984-884f-50f8d2144559" containerID="1560f92745fe53548214df6cb0904495acd37d96e0a81b5b4323c6773433b8e3" exitCode=0
Oct 30 19:20:46.891190 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.891146    4712 manager.go:324] Recovery completed
Oct 30 19:20:46.908625 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.908603    4712 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:20:46.908625 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.908618    4712 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:20:46.908728 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.908633    4712 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:20:46.908758 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.908738    4712 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 19:20:46.908786 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.908748    4712 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 19:20:46.908786 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.908765    4712 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 19:20:46.908786 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.908770    4712 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 19:20:46.908786 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.908775    4712 policy_none.go:50] "Start"
Oct 30 19:20:46.908786 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.908784    4712 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:20:46.908919 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.908792    4712 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:20:46.908919 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.908907    4712 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 19:20:46.908919 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.908913    4712 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 19:20:46.908919 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.908918    4712 policy_none.go:44] "Start"
Oct 30 19:20:46.913302 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.912882    4712 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:20:46.918767 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.918752    4712 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:20:46.918858 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:20:46.918782    4712 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:20:46.918858 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.918793    4712 server.go:87] "Starting device plugin registration server"
Oct 30 19:20:46.919082 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.919069    4712 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:20:46.919151 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.919082    4712 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:20:46.919201 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.919171    4712 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:20:46.919293 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.919266    4712 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:20:46.919348 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.919301    4712 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:20:46.919348 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.919308    4712 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:20:46.919479 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.919342    4712 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:20:46.919290844Z","UUID":"1861729b-1e07-4de5-88f5-64c336a07b09","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:20:46.920481 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:20:46.920453    4712 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:20:46.920656 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.920636    4712 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:20:46.920765 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.920662    4712 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:20:46.945066 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.945031    4712 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal"]
Oct 30 19:20:46.945513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.945479    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal" event={"ID":"2670e93fcc64eb8a9f070c9bd973c8ad","Type":"ContainerStarted","Data":"466f6f568ef20c0f46bde94294aea51f826c1fff4167a1e9c1aac92e63ba5bc5"}
Oct 30 19:20:46.945694 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.945519    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal" event={"ID":"2670e93fcc64eb8a9f070c9bd973c8ad","Type":"ContainerDied","Data":"11a15e06b0d4eb386807e9a1fdf660153c2f718eca5719db9b858af0e675796c"}
Oct 30 19:20:46.945694 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.945539    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal" event={"ID":"2670e93fcc64eb8a9f070c9bd973c8ad","Type":"ContainerStarted","Data":"0008bcf05fe731a028bb8218b6d25cc4481c6f5e08353eed82e64ec4cf016693"}
Oct 30 19:20:46.945694 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.945548    4712 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:46.945694 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.945557    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal" event={"ID":"4e9154b2294c90c44348210589c7863a","Type":"ContainerStarted","Data":"3b706d55464f0ae0908571c9c2c821e2370728ba1e55ac42e5ff2ef1be76d1d2"}
Oct 30 19:20:46.945694 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.945573    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal" event={"ID":"4e9154b2294c90c44348210589c7863a","Type":"ContainerStarted","Data":"8bc6f2de22212ef491069993807ee1b09446c52f5dee4c880b8e8f28a5c4cb40"}
Oct 30 19:20:46.945694 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.945665    4712 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:46.976706 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.976688    4712 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:20:46.976784 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:20:46.976722    4712 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:46.977311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:46.977297    4712 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:20:46.977346 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:20:46.977334    4712 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-248.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:47.019802 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.019783    4712 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:20:47.020845 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.020830    4712 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 19:20:47.020900 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.020856    4712 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 19:20:47.020900 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.020866    4712 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-248.ec2.internal" event="NodeHasSufficientPID"
Oct 30 19:20:47.020900 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.020893    4712 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-248.ec2.internal"
Oct 30 19:20:47.051749 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.051728    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2670e93fcc64eb8a9f070c9bd973c8ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal\" (UID: \"2670e93fcc64eb8a9f070c9bd973c8ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:47.051828 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.051752    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e9154b2294c90c44348210589c7863a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-248.ec2.internal\" (UID: \"4e9154b2294c90c44348210589c7863a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:47.051828 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.051776    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2670e93fcc64eb8a9f070c9bd973c8ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal\" (UID: \"2670e93fcc64eb8a9f070c9bd973c8ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:47.057303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.057286    4712 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-248.ec2.internal"
Oct 30 19:20:47.057394 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.057330    4712 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-248.ec2.internal"
Oct 30 19:20:47.152872 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.152818    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e9154b2294c90c44348210589c7863a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-248.ec2.internal\" (UID: \"4e9154b2294c90c44348210589c7863a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:47.152872 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.152849    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2670e93fcc64eb8a9f070c9bd973c8ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal\" (UID: \"2670e93fcc64eb8a9f070c9bd973c8ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:47.153007 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.152875    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2670e93fcc64eb8a9f070c9bd973c8ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal\" (UID: \"2670e93fcc64eb8a9f070c9bd973c8ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:47.153007 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.152909    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2670e93fcc64eb8a9f070c9bd973c8ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal\" (UID: \"2670e93fcc64eb8a9f070c9bd973c8ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:47.153007 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.152933    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2670e93fcc64eb8a9f070c9bd973c8ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal\" (UID: \"2670e93fcc64eb8a9f070c9bd973c8ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:47.153007 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.152935    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e9154b2294c90c44348210589c7863a-config\") pod \"kube-apiserver-proxy-ip-10-0-140-248.ec2.internal\" (UID: \"4e9154b2294c90c44348210589c7863a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:47.788830 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.788804    4712 apiserver.go:52] "Watching apiserver"
Oct 30 19:20:47.829239 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.829218    4712 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:20:47.829705 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.829683    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bm87v","openshift-kube-storage-version-migrator/migrator-66df5d496b-nxtzp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal","openshift-network-operator/iptables-alerter-lqm86","kube-system/konnectivity-agent-l65sf","openshift-dns/node-resolver-n9frl","openshift-multus/multus-additional-cni-plugins-jjm2b","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n","openshift-insights/insights-runtime-extractor-dxp6q","openshift-network-diagnostics/network-check-target-6dnzx","openshift-ovn-kubernetes/ovnkube-node-6rkvc","openshift-multus/multus-bw8v2","openshift-multus/network-metrics-daemon-vcr5t","openshift-image-registry/node-ca-qrsz7","kube-system/global-pull-secret-syncer-27xsq","openshift-insights/periodic-gathering-snzrt-w5xjz","openshift-ingress-canary/ingress-canary-bz68x","kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal","openshift-cluster-node-tuning-operator/tuned-2gf9g","openshift-service-ca/service-ca-5fffc97bdf-c49qz"]
Oct 30 19:20:47.836869 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.836840    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:47.852132 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.852108    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bkrzr\""
Oct 30 19:20:47.852671 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.852647    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:20:47.852772 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.852679    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9smtf\""
Oct 30 19:20:47.852772 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.852763    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8rhbj\""
Oct 30 19:20:47.852881 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.852805    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:20:47.852931 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.852881    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:20:47.852978 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.852960    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bcv2g\""
Oct 30 19:20:47.852978 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.852960    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-w8vq8\""
Oct 30 19:20:47.853065 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.852964    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gx26w\""
Oct 30 19:20:47.853065 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.853022    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b7kzg\""
Oct 30 19:20:47.853065 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.853054    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:20:47.853251 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.853100    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:20:47.853251 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.853130    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hxbtq\""
Oct 30 19:20:47.853251 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.853155    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:20:47.853251 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.853190    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:20:47.864720 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.864701    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:20:47.864821 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.864723    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:20:47.864821 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.864747    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:20:47.864821 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.864764    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:20:47.864821 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.864776    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:20:47.864821 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.864812    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:20:47.865007 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.864826    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:20:47.865007 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.864835    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:20:47.865007 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.864845    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:20:47.865007 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.864851    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:20:47.865007 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.864853    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:20:47.865007 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.864847    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:20:47.865007 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.864900    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:20:47.865007 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.864958    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sl59z\""
Oct 30 19:20:47.865317 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865078    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:20:47.865317 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865082    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:20:47.865317 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865108    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:20:47.865317 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865254    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:20:47.865317 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865269    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:20:47.865317 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865281    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:20:47.865317 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865320    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:20:47.865663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865352    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m2qt2\""
Oct 30 19:20:47.865663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865390    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:20:47.865663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865414    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-65nsn\""
Oct 30 19:20:47.865663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865467    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:20:47.865663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865481    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5ffd5\""
Oct 30 19:20:47.865663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865486    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-brmvf\""
Oct 30 19:20:47.865663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865504    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gx5x8\""
Oct 30 19:20:47.865663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865562    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lkdbb\""
Oct 30 19:20:47.865663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865572    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:20:47.865663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865591    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-77hs9\""
Oct 30 19:20:47.865663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865612    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 19:20:47.865663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865640    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hh67w\""
Oct 30 19:20:47.865663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865647    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:20:47.865663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865647    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 19:20:47.865663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865657    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:20:47.866214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865762    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qmnn8\""
Oct 30 19:20:47.866214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865822    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 19:20:47.866214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865827    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:20:47.866214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865836    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 19:20:47.866214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865868    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:20:47.866214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865911    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:20:47.866214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865914    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 19:20:47.866214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865924    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:20:47.866214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865926    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:20:47.866214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.865964    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:20:47.866214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.866032    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 19:20:47.866214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.866135    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:20:47.866214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.866168    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 19:20:47.866214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.866204    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:20:47.877604 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.877579    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 19:20:47.890699 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.890676    4712 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:47.890839 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.890825    4712 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:47.898390 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.898346    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:47.911437 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.911414    4712 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:20:47.915511 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.915497    4712 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:20:47.915572 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:20:47.915536    4712 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-248.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:47.916150 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.916135    4712 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:20:47.916208 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:20:47.916175    4712 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-248.ec2.internal"
Oct 30 19:20:47.956509 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.956491    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9838f46c-794d-4b6f-9f2d-a1a45e21819f-archives-path\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:47.956579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.956517    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-node-log\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.956579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.956532    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-cni-netd\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.956579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.956548    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aacc7bb9-0109-4984-884f-50f8d2144559-cni-binary-copy\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:47.956579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.956564    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b45e511-0890-4a70-96d7-01ba249d65cb-cni-binary-copy\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.956729 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.956601    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-sysconfig\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:47.956729 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.956710    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9838f46c-794d-4b6f-9f2d-a1a45e21819f-archives-path\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:47.956800 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.956787    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8c299f86-7f6e-4666-ab9c-542537e6ff8e-env-overrides\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.956859 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.956845    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90c7079a-e03a-4d3e-bb41-b61ce36292ec-hosts-file\") pod \"node-resolver-n9frl\" (UID: \"90c7079a-e03a-4d3e-bb41-b61ce36292ec\") " pod="openshift-dns/node-resolver-n9frl"
Oct 30 19:20:47.956896 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.956872    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-var-lib-cni-bin\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.956896 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.956876    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b45e511-0890-4a70-96d7-01ba249d65cb-cni-binary-copy\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.956896 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.956886    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aacc7bb9-0109-4984-884f-50f8d2144559-cni-binary-copy\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:47.956993 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.956889    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-var-lib-cni-multus\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.956993 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.956933    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-var-lib-kubelet\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.956993 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.956956    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8c299f86-7f6e-4666-ab9c-542537e6ff8e-env-overrides\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.956993 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.956972    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aacc7bb9-0109-4984-884f-50f8d2144559-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:47.957104 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.956997    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-run-openvswitch\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.957104 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957016    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d98ps\" (UniqueName: \"kubernetes.io/projected/7948dcd5-eb52-4510-a73d-25b869e1394e-kube-api-access-d98ps\") pod \"migrator-66df5d496b-nxtzp\" (UID: \"7948dcd5-eb52-4510-a73d-25b869e1394e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxtzp"
Oct 30 19:20:47.957104 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957032    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-trusted-ca-bundle\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:47.957104 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957059    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8cf3812-0483-481a-86e6-8c20c76d3426-cert\") pod \"ingress-canary-bz68x\" (UID: \"e8cf3812-0483-481a-86e6-8c20c76d3426\") " pod="openshift-ingress-canary/ingress-canary-bz68x"
Oct 30 19:20:47.957226 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957135    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aacc7bb9-0109-4984-884f-50f8d2144559-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:47.957226 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957141    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-log-socket\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.957226 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957179    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f29fd70f-b10e-4154-8652-6b17fbe3cf13-host-slash\") pod \"iptables-alerter-lqm86\" (UID: \"f29fd70f-b10e-4154-8652-6b17fbe3cf13\") " pod="openshift-network-operator/iptables-alerter-lqm86"
Oct 30 19:20:47.957226 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957212    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-systemd\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:47.957415 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957238    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8cf3812-0483-481a-86e6-8c20c76d3426-cert\") pod \"ingress-canary-bz68x\" (UID: \"e8cf3812-0483-481a-86e6-8c20c76d3426\") " pod="openshift-ingress-canary/ingress-canary-bz68x"
Oct 30 19:20:47.957415 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957253    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdmgh\" (UniqueName: \"kubernetes.io/projected/9838f46c-794d-4b6f-9f2d-a1a45e21819f-kube-api-access-kdmgh\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:47.957415 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957306    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-trusted-ca-bundle\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:47.957415 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957409    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-cni-dir\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.957581 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957426    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-run-multus-certs\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.957581 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957441    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-sys\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:47.957581 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957455    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-host\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:47.957581 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957472    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kpwc\" (UniqueName: \"kubernetes.io/projected/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-kube-api-access-5kpwc\") pod \"network-metrics-daemon-vcr5t\" (UID: \"029fdc09-aaca-4dc2-90ff-86e05c26bf9e\") " pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:47.957581 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957504    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9zdg\" (UniqueName: \"kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg\") pod \"network-check-target-6dnzx\" (UID: \"c1a81d1f-b828-4a0d-a57a-df3d493947d5\") " pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:47.957581 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957524    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-slash\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.957581 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957538    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-run-netns\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.957581 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957552    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-run-ovn\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.957939 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957601    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-run-netns\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.957939 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957639    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-run\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:47.957939 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957770    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-etc-openvswitch\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.957939 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957790    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/954bf0fe-4287-4435-9abb-fd06dc478099-agent-certs\") pod \"konnectivity-agent-l65sf\" (UID: \"954bf0fe-4287-4435-9abb-fd06dc478099\") " pod="kube-system/konnectivity-agent-l65sf"
Oct 30 19:20:47.957939 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957808    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:47.957939 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957838    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-kubelet\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.957939 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957867    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aacc7bb9-0109-4984-884f-50f8d2144559-system-cni-dir\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:47.957939 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957895    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbf67380-1538-474c-ad65-4f8194a3d0af-data-volume\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:47.957939 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.957920    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:47.958303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958022    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-sysctl-d\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:47.958303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958058    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbf67380-1538-474c-ad65-4f8194a3d0af-data-volume\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:47.958303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958071    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.958303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958099    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbd278f9-76f5-41ed-93d1-435804741834-original-pull-secret\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:47.958303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958101    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90c7079a-e03a-4d3e-bb41-b61ce36292ec-tmp-dir\") pod \"node-resolver-n9frl\" (UID: \"90c7079a-e03a-4d3e-bb41-b61ce36292ec\") " pod="openshift-dns/node-resolver-n9frl"
Oct 30 19:20:47.958303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958148    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drlzm\" (UniqueName: \"kubernetes.io/projected/aacc7bb9-0109-4984-884f-50f8d2144559-kube-api-access-drlzm\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:47.958303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958161    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90c7079a-e03a-4d3e-bb41-b61ce36292ec-tmp-dir\") pod \"node-resolver-n9frl\" (UID: \"90c7079a-e03a-4d3e-bb41-b61ce36292ec\") " pod="openshift-dns/node-resolver-n9frl"
Oct 30 19:20:47.958303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958105    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/954bf0fe-4287-4435-9abb-fd06dc478099-agent-certs\") pod \"konnectivity-agent-l65sf\" (UID: \"954bf0fe-4287-4435-9abb-fd06dc478099\") " pod="kube-system/konnectivity-agent-l65sf"
Oct 30 19:20:47.958303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958178    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-sys-fs\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:47.958303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958173    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:47.958303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958206    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-run-k8s-cni-cncf-io\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.958303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958226    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqxbc\" (UniqueName: \"kubernetes.io/projected/0b45e511-0890-4a70-96d7-01ba249d65cb-kube-api-access-lqxbc\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.958303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958245    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0cffcb83-c327-45ab-90b7-837f3f455f92-signing-key\") pod \"service-ca-5fffc97bdf-c49qz\" (UID: \"0cffcb83-c327-45ab-90b7-837f3f455f92\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz"
Oct 30 19:20:47.958303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958273    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-tuned\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:47.958303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958297    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b368ff6-93f5-472f-97ae-3f9ffee9102d-tmp-dir\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958321    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-run-ovn-kubernetes\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958345    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-cni-bin\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958393    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8c299f86-7f6e-4666-ab9c-542537e6ff8e-ovnkube-script-lib\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958403    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-tuned\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958431    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958442    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b368ff6-93f5-472f-97ae-3f9ffee9102d-tmp-dir\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958505    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-registration-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958506    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0cffcb83-c327-45ab-90b7-837f3f455f92-signing-key\") pod \"service-ca-5fffc97bdf-c49qz\" (UID: \"0cffcb83-c327-45ab-90b7-837f3f455f92\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958534    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spzw5\" (UniqueName: \"kubernetes.io/projected/f29fd70f-b10e-4154-8652-6b17fbe3cf13-kube-api-access-spzw5\") pod \"iptables-alerter-lqm86\" (UID: \"f29fd70f-b10e-4154-8652-6b17fbe3cf13\") " pod="openshift-network-operator/iptables-alerter-lqm86"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958569    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0cffcb83-c327-45ab-90b7-837f3f455f92-signing-cabundle\") pod \"service-ca-5fffc97bdf-c49qz\" (UID: \"0cffcb83-c327-45ab-90b7-837f3f455f92\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958602    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8c299f86-7f6e-4666-ab9c-542537e6ff8e-ovnkube-script-lib\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958609    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f29fd70f-b10e-4154-8652-6b17fbe3cf13-iptables-alerter-script\") pod \"iptables-alerter-lqm86\" (UID: \"f29fd70f-b10e-4154-8652-6b17fbe3cf13\") " pod="openshift-network-operator/iptables-alerter-lqm86"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958638    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-system-cni-dir\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958654    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-kubernetes\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958670    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-run-systemd\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958689    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b368ff6-93f5-472f-97ae-3f9ffee9102d-metrics-tls\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958722    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0cffcb83-c327-45ab-90b7-837f3f455f92-signing-cabundle\") pod \"service-ca-5fffc97bdf-c49qz\" (UID: \"0cffcb83-c327-45ab-90b7-837f3f455f92\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958724    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-systemd-units\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.958768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958763    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c299f86-7f6e-4666-ab9c-542537e6ff8e-ovn-node-metrics-cert\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958787    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-hostroot\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958884    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f29fd70f-b10e-4154-8652-6b17fbe3cf13-iptables-alerter-script\") pod \"iptables-alerter-lqm86\" (UID: \"f29fd70f-b10e-4154-8652-6b17fbe3cf13\") " pod="openshift-network-operator/iptables-alerter-lqm86"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958906    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b368ff6-93f5-472f-97ae-3f9ffee9102d-metrics-tls\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958910    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs\") pod \"network-metrics-daemon-vcr5t\" (UID: \"029fdc09-aaca-4dc2-90ff-86e05c26bf9e\") " pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958926    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aacc7bb9-0109-4984-884f-50f8d2144559-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958951    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-etc-selinux\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958974    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/954bf0fe-4287-4435-9abb-fd06dc478099-konnectivity-ca\") pod \"konnectivity-agent-l65sf\" (UID: \"954bf0fe-4287-4435-9abb-fd06dc478099\") " pod="kube-system/konnectivity-agent-l65sf"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.958990    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk84z\" (UniqueName: \"kubernetes.io/projected/e8cf3812-0483-481a-86e6-8c20c76d3426-kube-api-access-hk84z\") pod \"ingress-canary-bz68x\" (UID: \"e8cf3812-0483-481a-86e6-8c20c76d3426\") " pod="openshift-ingress-canary/ingress-canary-bz68x"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959016    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aacc7bb9-0109-4984-884f-50f8d2144559-cnibin\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959030    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-cnibin\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959044    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-socket-dir-parent\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959060    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8c299f86-7f6e-4666-ab9c-542537e6ff8e-ovn-node-metrics-cert\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959061    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv6g9\" (UniqueName: \"kubernetes.io/projected/90c7079a-e03a-4d3e-bb41-b61ce36292ec-kube-api-access-pv6g9\") pod \"node-resolver-n9frl\" (UID: \"90c7079a-e03a-4d3e-bb41-b61ce36292ec\") " pod="openshift-dns/node-resolver-n9frl"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959093    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbf67380-1538-474c-ad65-4f8194a3d0af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959111    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9txd\" (UniqueName: \"kubernetes.io/projected/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-api-access-s9txd\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959127    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada-serviceca\") pod \"node-ca-qrsz7\" (UID: \"a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada\") " pod="openshift-image-registry/node-ca-qrsz7"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959142    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-etc-kubernetes\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959157    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-modprobe-d\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:47.959314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959172    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8c299f86-7f6e-4666-ab9c-542537e6ff8e-ovnkube-config\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959326    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8c299f86-7f6e-4666-ab9c-542537e6ff8e-ovnkube-config\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959329    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada-serviceca\") pod \"node-ca-qrsz7\" (UID: \"a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada\") " pod="openshift-image-registry/node-ca-qrsz7"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959352    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aacc7bb9-0109-4984-884f-50f8d2144559-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959388    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-socket-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959402    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cbf67380-1538-474c-ad65-4f8194a3d0af-crio-socket\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959417    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada-host\") pod \"node-ca-qrsz7\" (UID: \"a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada\") " pod="openshift-image-registry/node-ca-qrsz7"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959434    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bf6h\" (UniqueName: \"kubernetes.io/projected/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada-kube-api-access-4bf6h\") pod \"node-ca-qrsz7\" (UID: \"a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada\") " pod="openshift-image-registry/node-ca-qrsz7"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959462    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwxjt\" (UniqueName: \"kubernetes.io/projected/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-kube-api-access-gwxjt\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959481    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-service-ca-bundle\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959496    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b368ff6-93f5-472f-97ae-3f9ffee9102d-config-volume\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959546    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-daemon-config\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959563    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ll8f\" (UniqueName: \"kubernetes.io/projected/0cffcb83-c327-45ab-90b7-837f3f455f92-kube-api-access-5ll8f\") pod \"service-ca-5fffc97bdf-c49qz\" (UID: \"0cffcb83-c327-45ab-90b7-837f3f455f92\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959577    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbd278f9-76f5-41ed-93d1-435804741834-dbus\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959596    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwzm8\" (UniqueName: \"kubernetes.io/projected/1b368ff6-93f5-472f-97ae-3f9ffee9102d-kube-api-access-nwzm8\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959598    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/954bf0fe-4287-4435-9abb-fd06dc478099-konnectivity-ca\") pod \"konnectivity-agent-l65sf\" (UID: \"954bf0fe-4287-4435-9abb-fd06dc478099\") " pod="kube-system/konnectivity-agent-l65sf"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959613    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-device-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959629    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-conf-dir\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959645    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-sysctl-conf\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959658    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-tmp\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959674    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbd278f9-76f5-41ed-93d1-435804741834-kubelet-config\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959691    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-var-lib-openvswitch\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959767    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-tmp\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959766    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5xl5\" (UniqueName: \"kubernetes.io/projected/8c299f86-7f6e-4666-ab9c-542537e6ff8e-kube-api-access-d5xl5\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959793    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkgsr\" (UniqueName: \"kubernetes.io/projected/093cd549-ebec-45f5-b888-42019476c70c-kube-api-access-tkgsr\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959812    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-lib-modules\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:47.959875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959828    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-var-lib-kubelet\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:47.960555 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.959896    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b368ff6-93f5-472f-97ae-3f9ffee9102d-config-volume\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:47.960555 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.960078    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aacc7bb9-0109-4984-884f-50f8d2144559-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:47.960555 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.960328    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbf67380-1538-474c-ad65-4f8194a3d0af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:47.960555 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.960458    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-service-ca-bundle\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:47.960555 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.960486    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-daemon-config\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.976253 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.976231    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d98ps\" (UniqueName: \"kubernetes.io/projected/7948dcd5-eb52-4510-a73d-25b869e1394e-kube-api-access-d98ps\") pod \"migrator-66df5d496b-nxtzp\" (UID: \"7948dcd5-eb52-4510-a73d-25b869e1394e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nxtzp"
Oct 30 19:20:47.982750 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.982730    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwzm8\" (UniqueName: \"kubernetes.io/projected/1b368ff6-93f5-472f-97ae-3f9ffee9102d-kube-api-access-nwzm8\") pod \"dns-default-bm87v\" (UID: \"1b368ff6-93f5-472f-97ae-3f9ffee9102d\") " pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:47.983155 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.983139    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdmgh\" (UniqueName: \"kubernetes.io/projected/9838f46c-794d-4b6f-9f2d-a1a45e21819f-kube-api-access-kdmgh\") pod \"periodic-gathering-snzrt-w5xjz\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") " pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:20:47.985253 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.985232    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkgsr\" (UniqueName: \"kubernetes.io/projected/093cd549-ebec-45f5-b888-42019476c70c-kube-api-access-tkgsr\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:47.985320 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.985307    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kpwc\" (UniqueName: \"kubernetes.io/projected/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-kube-api-access-5kpwc\") pod \"network-metrics-daemon-vcr5t\" (UID: \"029fdc09-aaca-4dc2-90ff-86e05c26bf9e\") " pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:47.987825 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.987805    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drlzm\" (UniqueName: \"kubernetes.io/projected/aacc7bb9-0109-4984-884f-50f8d2144559-kube-api-access-drlzm\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:47.988007 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.987992    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwxjt\" (UniqueName: \"kubernetes.io/projected/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-kube-api-access-gwxjt\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:47.988108 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.988092    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9zdg\" (UniqueName: \"kubernetes.io/projected/c1a81d1f-b828-4a0d-a57a-df3d493947d5-kube-api-access-r9zdg\") pod \"network-check-target-6dnzx\" (UID: \"c1a81d1f-b828-4a0d-a57a-df3d493947d5\") " pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:47.989414 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.989340    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spzw5\" (UniqueName: \"kubernetes.io/projected/f29fd70f-b10e-4154-8652-6b17fbe3cf13-kube-api-access-spzw5\") pod \"iptables-alerter-lqm86\" (UID: \"f29fd70f-b10e-4154-8652-6b17fbe3cf13\") " pod="openshift-network-operator/iptables-alerter-lqm86"
Oct 30 19:20:47.989414 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.989376    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9txd\" (UniqueName: \"kubernetes.io/projected/cbf67380-1538-474c-ad65-4f8194a3d0af-kube-api-access-s9txd\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:47.989555 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.989479    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv6g9\" (UniqueName: \"kubernetes.io/projected/90c7079a-e03a-4d3e-bb41-b61ce36292ec-kube-api-access-pv6g9\") pod \"node-resolver-n9frl\" (UID: \"90c7079a-e03a-4d3e-bb41-b61ce36292ec\") " pod="openshift-dns/node-resolver-n9frl"
Oct 30 19:20:47.989555 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.989479    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5xl5\" (UniqueName: \"kubernetes.io/projected/8c299f86-7f6e-4666-ab9c-542537e6ff8e-kube-api-access-d5xl5\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:47.989641 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.989627    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqxbc\" (UniqueName: \"kubernetes.io/projected/0b45e511-0890-4a70-96d7-01ba249d65cb-kube-api-access-lqxbc\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:47.989880 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.989861    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bf6h\" (UniqueName: \"kubernetes.io/projected/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada-kube-api-access-4bf6h\") pod \"node-ca-qrsz7\" (UID: \"a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada\") " pod="openshift-image-registry/node-ca-qrsz7"
Oct 30 19:20:47.990285 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.990270    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk84z\" (UniqueName: \"kubernetes.io/projected/e8cf3812-0483-481a-86e6-8c20c76d3426-kube-api-access-hk84z\") pod \"ingress-canary-bz68x\" (UID: \"e8cf3812-0483-481a-86e6-8c20c76d3426\") " pod="openshift-ingress-canary/ingress-canary-bz68x"
Oct 30 19:20:47.991060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:47.991042    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ll8f\" (UniqueName: \"kubernetes.io/projected/0cffcb83-c327-45ab-90b7-837f3f455f92-kube-api-access-5ll8f\") pod \"service-ca-5fffc97bdf-c49qz\" (UID: \"0cffcb83-c327-45ab-90b7-837f3f455f92\") " pod="openshift-service-ca/service-ca-5fffc97bdf-c49qz"
Oct 30 19:20:48.060307 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060254    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbd278f9-76f5-41ed-93d1-435804741834-dbus\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:48.060307 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060282    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-device-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:48.060307 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060298    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-conf-dir\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.060503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060318    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-sysctl-conf\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.060503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060343    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbd278f9-76f5-41ed-93d1-435804741834-kubelet-config\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:48.060503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060383    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-var-lib-openvswitch\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.060503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060388    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-device-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:48.060503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060411    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbd278f9-76f5-41ed-93d1-435804741834-dbus\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:48.060503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060423    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-lib-modules\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.060503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060433    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-var-lib-openvswitch\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.060503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060447    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-var-lib-kubelet\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.060503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060419    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-conf-dir\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.060503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060470    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-node-log\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.060503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060484    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbd278f9-76f5-41ed-93d1-435804741834-kubelet-config\") pod \"global-pull-secret-syncer-27xsq\" (UID: \"bbd278f9-76f5-41ed-93d1-435804741834\") " pod="kube-system/global-pull-secret-syncer-27xsq"
Oct 30 19:20:48.060503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060495    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-cni-netd\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.060503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060506    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-lib-modules\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.060503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060508    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-var-lib-kubelet\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060508    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-sysctl-conf\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060519    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-sysconfig\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060545    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-cni-netd\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060539    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-node-log\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060563    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-sysconfig\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060573    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90c7079a-e03a-4d3e-bb41-b61ce36292ec-hosts-file\") pod \"node-resolver-n9frl\" (UID: \"90c7079a-e03a-4d3e-bb41-b61ce36292ec\") " pod="openshift-dns/node-resolver-n9frl"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060598    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-var-lib-cni-bin\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060622    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-var-lib-cni-multus\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060635    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90c7079a-e03a-4d3e-bb41-b61ce36292ec-hosts-file\") pod \"node-resolver-n9frl\" (UID: \"90c7079a-e03a-4d3e-bb41-b61ce36292ec\") " pod="openshift-dns/node-resolver-n9frl"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060656    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-var-lib-kubelet\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060679    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-run-openvswitch\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060683    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-var-lib-cni-multus\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060685    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-var-lib-cni-bin\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060702    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-log-socket\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060709    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-run-openvswitch\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060707    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-var-lib-kubelet\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060727    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f29fd70f-b10e-4154-8652-6b17fbe3cf13-host-slash\") pod \"iptables-alerter-lqm86\" (UID: \"f29fd70f-b10e-4154-8652-6b17fbe3cf13\") " pod="openshift-network-operator/iptables-alerter-lqm86"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060753    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-systemd\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060777    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-cni-dir\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060786    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f29fd70f-b10e-4154-8652-6b17fbe3cf13-host-slash\") pod \"iptables-alerter-lqm86\" (UID: \"f29fd70f-b10e-4154-8652-6b17fbe3cf13\") " pod="openshift-network-operator/iptables-alerter-lqm86"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060800    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-run-multus-certs\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060815    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-cni-dir\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060819    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-systemd\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060822    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-run-multus-certs\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060781    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-log-socket\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060823    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-sys\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060859    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-sys\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060872    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-host\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060897    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-slash\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060920    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-host\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060920    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-run-netns\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060942    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-slash\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060950    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-run-netns\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060957    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-run-ovn\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060979    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-run-ovn\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.060985    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-run-netns\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061008    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-run\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061030    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-etc-openvswitch\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061039    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-run-netns\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061054    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-kubelet\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061077    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-run\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.061061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061077    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aacc7bb9-0109-4984-884f-50f8d2144559-system-cni-dir\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061104    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-sysctl-d\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061110    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aacc7bb9-0109-4984-884f-50f8d2144559-system-cni-dir\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061119    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-etc-openvswitch\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061125    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061127    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-kubelet\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061153    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061156    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-sys-fs\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061180    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-run-k8s-cni-cncf-io\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061200    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-host-run-k8s-cni-cncf-io\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061202    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-run-ovn-kubernetes\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061205    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-sysctl-d\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061182    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-sys-fs\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061220    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-run-ovn-kubernetes\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061221    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-cni-bin\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061249    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-host-cni-bin\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061250    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061276    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061283    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-registration-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061309    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-system-cni-dir\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061312    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-registration-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061330    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-kubernetes\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061352    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-run-systemd\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061378    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-system-cni-dir\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061399    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-systemd-units\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061422    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-hostroot\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061429    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-kubernetes\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061443    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-systemd-units\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061445    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs\") pod \"network-metrics-daemon-vcr5t\" (UID: \"029fdc09-aaca-4dc2-90ff-86e05c26bf9e\") " pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061421    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8c299f86-7f6e-4666-ab9c-542537e6ff8e-run-systemd\") pod \"ovnkube-node-6rkvc\" (UID: \"8c299f86-7f6e-4666-ab9c-542537e6ff8e\") " pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061481    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aacc7bb9-0109-4984-884f-50f8d2144559-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061505    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-etc-selinux\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061527    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aacc7bb9-0109-4984-884f-50f8d2144559-cnibin\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061529    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aacc7bb9-0109-4984-884f-50f8d2144559-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061473    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-hostroot\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061550    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-cnibin\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061575    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-socket-dir-parent\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061598    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-etc-selinux\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061608    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-etc-kubernetes\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061610    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aacc7bb9-0109-4984-884f-50f8d2144559-cnibin\") pod \"multus-additional-cni-plugins-jjm2b\" (UID: \"aacc7bb9-0109-4984-884f-50f8d2144559\") " pod="openshift-multus/multus-additional-cni-plugins-jjm2b"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061636    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-modprobe-d\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061689    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-multus-socket-dir-parent\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061701    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-socket-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061722    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4aa3418-bbc0-4c70-a42a-2a65e478b47d-etc-modprobe-d\") pod \"tuned-2gf9g\" (UID: \"c4aa3418-bbc0-4c70-a42a-2a65e478b47d\") " pod="openshift-cluster-node-tuning-operator/tuned-2gf9g"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061729    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cbf67380-1538-474c-ad65-4f8194a3d0af-crio-socket\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061736    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-etc-kubernetes\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061641    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b45e511-0890-4a70-96d7-01ba249d65cb-cnibin\") pod \"multus-bw8v2\" (UID: \"0b45e511-0890-4a70-96d7-01ba249d65cb\") " pod="openshift-multus/multus-bw8v2"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061766    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/093cd549-ebec-45f5-b888-42019476c70c-socket-dir\") pod \"aws-ebs-csi-driver-node-nqt7n\" (UID: \"093cd549-ebec-45f5-b888-42019476c70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nqt7n"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061770    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada-host\") pod \"node-ca-qrsz7\" (UID: \"a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada\") " pod="openshift-image-registry/node-ca-qrsz7"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061766    4712 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061803    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cbf67380-1538-474c-ad65-4f8194a3d0af-crio-socket\") pod \"insights-runtime-extractor-dxp6q\" (UID: \"cbf67380-1538-474c-ad65-4f8194a3d0af\") " pod="openshift-insights/insights-runtime-extractor-dxp6q"
Oct 30 19:20:48.062674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.061801    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada-host\") pod \"node-ca-qrsz7\" (UID: \"a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada\") " pod="openshift-image-registry/node-ca-qrsz7"
Oct 30 19:20:48.064308 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.063891    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/029fdc09-aaca-4dc2-90ff-86e05c26bf9e-metrics-certs\") pod \"network-metrics-daemon-vcr5t\" (UID: \"029fdc09-aaca-4dc2-90ff-86e05c26bf9e\") " pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:48.121901 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.121877    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:48.138143 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.138125    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vcr5t"
Oct 30 19:20:48.139648 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.139631    4712 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:20:48.453253 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.453233    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vcr5t"]
Oct 30 19:20:48.454951 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:48.454919    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029fdc09_aaca_4dc2_90ff_86e05c26bf9e.slice/crio-fcf19fba0de806682ff1a77a583d34b7d2f585d0133b52ad0a2b1f3df9220622 WatchSource:0}: Error finding container fcf19fba0de806682ff1a77a583d34b7d2f585d0133b52ad0a2b1f3df9220622: Status 404 returned error can't find the container with id fcf19fba0de806682ff1a77a583d34b7d2f585d0133b52ad0a2b1f3df9220622
Oct 30 19:20:48.630983 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.630958    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:48.897124 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.897049    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vcr5t" event={"ID":"029fdc09-aaca-4dc2-90ff-86e05c26bf9e","Type":"ContainerStarted","Data":"fcf19fba0de806682ff1a77a583d34b7d2f585d0133b52ad0a2b1f3df9220622"}
Oct 30 19:20:48.942195 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:48.942169    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l65sf"
Oct 30 19:20:49.289206 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:49.289129    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:49.441534 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:49.441493    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:49.590602 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:49.590503    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:49.749542 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:49.749516    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:49.901843 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:49.901818    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dxp6q" event={"ID":"cbf67380-1538-474c-ad65-4f8194a3d0af","Type":"ContainerStarted","Data":"2448f610aabb1dd15d03f01d10e23c637d8d49a16804cb7cbf9742a2ca025085"}
Oct 30 19:20:49.903630 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:49.903608    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vcr5t" event={"ID":"029fdc09-aaca-4dc2-90ff-86e05c26bf9e","Type":"ContainerStarted","Data":"04d808337c5038b745c4fb59ade80b1fbdb8d9790f6ee775523069edbbdec3d7"}
Oct 30 19:20:49.903732 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:49.903648    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vcr5t" event={"ID":"029fdc09-aaca-4dc2-90ff-86e05c26bf9e","Type":"ContainerStarted","Data":"a333af6d6191508479937fcc8f9be473a87390a081c885a29f40d626ad851109"}
Oct 30 19:20:49.917444 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:49.917423    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:50.101951 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:50.101922    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:50.463090 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:50.463063    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:50.603411 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:50.603387    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:50.814490 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:50.814426    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:51.308076 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:51.308041    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:51.445472 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:51.445449    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:51.577174 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:51.577116    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:51.762509 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:51.762485    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:51.928912 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:51.928886    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:52.100310 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:52.100278    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:52.197084 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:52.197026    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:20:52.254266 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:52.254243    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:52.556875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:52.556817    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:52.707068 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:52.707041    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:52.893409 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:52.893389    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:53.339540 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:53.339472    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:53.481921 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:53.481896    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:55.436953 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:55.436928    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:20:55.437419 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:55.437194    4712 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:20:57.898956 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:57.898927    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bm87v"
Oct 30 19:20:58.996978 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:58.996942    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r8gv2"]
Oct 30 19:20:58.999816 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:58.999793    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r8gv2"
Oct 30 19:20:59.009538 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.009517    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-hnr6l"]
Oct 30 19:20:59.012118 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.012102    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-hnr6l"
Oct 30 19:20:59.022010 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.021990    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 19:20:59.022101 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.022021    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gw7nc\""
Oct 30 19:20:59.027308 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.027284    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r8gv2"]
Oct 30 19:20:59.034598 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.034581    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 19:20:59.034693 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.034627    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 19:20:59.035143 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.035128    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-hnr6l"]
Oct 30 19:20:59.090640 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.090611    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:59.093240 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.093220    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7974c9c7fd-g85kb"]
Oct 30 19:20:59.095961 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.095946    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.128712 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.128690    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a6076fa2-677c-4d8e-871c-238ec152066c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-r8gv2\" (UID: \"a6076fa2-677c-4d8e-871c-238ec152066c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r8gv2"
Oct 30 19:20:59.128797 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.128719    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52a21ddd-696e-4d75-a0e0-424aba32607d-tmp\") pod \"downloads-6dd789c8b7-hnr6l\" (UID: \"52a21ddd-696e-4d75-a0e0-424aba32607d\") " pod="openshift-console/downloads-6dd789c8b7-hnr6l"
Oct 30 19:20:59.128849 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.128803    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pts4j\" (UniqueName: \"kubernetes.io/projected/52a21ddd-696e-4d75-a0e0-424aba32607d-kube-api-access-pts4j\") pod \"downloads-6dd789c8b7-hnr6l\" (UID: \"52a21ddd-696e-4d75-a0e0-424aba32607d\") " pod="openshift-console/downloads-6dd789c8b7-hnr6l"
Oct 30 19:20:59.134929 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.134909    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 19:20:59.135077 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.134956    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 19:20:59.135203 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.135080    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vp4s5\""
Oct 30 19:20:59.135203 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.135051    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-pn2s9\""
Oct 30 19:20:59.135203 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.135144    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 19:20:59.135465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.135444    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7974c9c7fd-g85kb"]
Oct 30 19:20:59.135827 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.135812    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 19:20:59.229111 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.229082    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s5kp\" (UniqueName: \"kubernetes.io/projected/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-kube-api-access-6s5kp\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.229231 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.229116    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52a21ddd-696e-4d75-a0e0-424aba32607d-tmp\") pod \"downloads-6dd789c8b7-hnr6l\" (UID: \"52a21ddd-696e-4d75-a0e0-424aba32607d\") " pod="openshift-console/downloads-6dd789c8b7-hnr6l"
Oct 30 19:20:59.229231 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.229136    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-registry-certificates\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.229231 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.229158    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-image-registry-private-configuration\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.229418 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.229395    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pts4j\" (UniqueName: \"kubernetes.io/projected/52a21ddd-696e-4d75-a0e0-424aba32607d-kube-api-access-pts4j\") pod \"downloads-6dd789c8b7-hnr6l\" (UID: \"52a21ddd-696e-4d75-a0e0-424aba32607d\") " pod="openshift-console/downloads-6dd789c8b7-hnr6l"
Oct 30 19:20:59.229478 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.229439    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a6076fa2-677c-4d8e-871c-238ec152066c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-r8gv2\" (UID: \"a6076fa2-677c-4d8e-871c-238ec152066c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r8gv2"
Oct 30 19:20:59.229478 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.229457    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-ca-trust-extracted\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.229478 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.229472    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-installation-pull-secrets\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.229626 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.229490    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-bound-sa-token\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.229626 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.229561    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52a21ddd-696e-4d75-a0e0-424aba32607d-tmp\") pod \"downloads-6dd789c8b7-hnr6l\" (UID: \"52a21ddd-696e-4d75-a0e0-424aba32607d\") " pod="openshift-console/downloads-6dd789c8b7-hnr6l"
Oct 30 19:20:59.229626 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.229619    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-registry-tls\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.229775 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.229643    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-trusted-ca\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.231852 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.231829    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a6076fa2-677c-4d8e-871c-238ec152066c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-r8gv2\" (UID: \"a6076fa2-677c-4d8e-871c-238ec152066c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r8gv2"
Oct 30 19:20:59.250956 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.250907    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pts4j\" (UniqueName: \"kubernetes.io/projected/52a21ddd-696e-4d75-a0e0-424aba32607d-kube-api-access-pts4j\") pod \"downloads-6dd789c8b7-hnr6l\" (UID: \"52a21ddd-696e-4d75-a0e0-424aba32607d\") " pod="openshift-console/downloads-6dd789c8b7-hnr6l"
Oct 30 19:20:59.286919 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.286881    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:59.311461 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.311440    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r8gv2"
Oct 30 19:20:59.323233 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.323214    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-hnr6l"
Oct 30 19:20:59.330236 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.330217    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-ca-trust-extracted\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.330340 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.330245    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-installation-pull-secrets\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.330340 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.330262    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-bound-sa-token\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.330340 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.330285    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-registry-tls\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.330340 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.330305    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-trusted-ca\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.330340 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.330336    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s5kp\" (UniqueName: \"kubernetes.io/projected/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-kube-api-access-6s5kp\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.330623 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.330389    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-registry-certificates\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.330623 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.330419    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-image-registry-private-configuration\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.330745 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.330728    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-ca-trust-extracted\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.331417 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.331392    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-registry-certificates\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.331580 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.331556    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-trusted-ca\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.333097 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.333068    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-installation-pull-secrets\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.333097 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.333086    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-registry-tls\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.333318 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.333233    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-image-registry-private-configuration\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.349987 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.349954    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-bound-sa-token\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.350546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.350522    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s5kp\" (UniqueName: \"kubernetes.io/projected/bcc2819e-b758-4cc7-b55b-b5fee59a83ff-kube-api-access-6s5kp\") pod \"image-registry-7974c9c7fd-g85kb\" (UID: \"bcc2819e-b758-4cc7-b55b-b5fee59a83ff\") " pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.406754 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.406668    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.425105 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.425078    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:59.559510 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.559440    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:20:59.604154 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.604132    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r8gv2"]
Oct 30 19:20:59.605733 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:59.605710    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6076fa2_677c_4d8e_871c_238ec152066c.slice/crio-d8b91cef41bee9f5e7d06d94a5ae04d53571b27b73061618d1bcdc3b4e6ad68e WatchSource:0}: Error finding container d8b91cef41bee9f5e7d06d94a5ae04d53571b27b73061618d1bcdc3b4e6ad68e: Status 404 returned error can't find the container with id d8b91cef41bee9f5e7d06d94a5ae04d53571b27b73061618d1bcdc3b4e6ad68e
Oct 30 19:20:59.637189 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.637171    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-hnr6l"]
Oct 30 19:20:59.638106 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:59.638086    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a21ddd_696e_4d75_a0e0_424aba32607d.slice/crio-12622e16fd9c0d2c74fa2b349cdb6876aae30e0dd998a8267ebf47627c4b29df WatchSource:0}: Error finding container 12622e16fd9c0d2c74fa2b349cdb6876aae30e0dd998a8267ebf47627c4b29df: Status 404 returned error can't find the container with id 12622e16fd9c0d2c74fa2b349cdb6876aae30e0dd998a8267ebf47627c4b29df
Oct 30 19:20:59.695799 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.695769    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7974c9c7fd-g85kb"]
Oct 30 19:20:59.699509 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:20:59.699485    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc2819e_b758_4cc7_b55b_b5fee59a83ff.slice/crio-bfcbd6794473dd5e0ce70b7e931ae1bc893eaf6e8830cc92eee7af804a8022b7 WatchSource:0}: Error finding container bfcbd6794473dd5e0ce70b7e931ae1bc893eaf6e8830cc92eee7af804a8022b7: Status 404 returned error can't find the container with id bfcbd6794473dd5e0ce70b7e931ae1bc893eaf6e8830cc92eee7af804a8022b7
Oct 30 19:20:59.935897 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.935857    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb" event={"ID":"bcc2819e-b758-4cc7-b55b-b5fee59a83ff","Type":"ContainerStarted","Data":"04db7dd84bae61c863c094f2486e3420c5b3c1a075ebaf8c18fdbaf059b77bd5"}
Oct 30 19:20:59.936085 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.935916    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb" event={"ID":"bcc2819e-b758-4cc7-b55b-b5fee59a83ff","Type":"ContainerStarted","Data":"bfcbd6794473dd5e0ce70b7e931ae1bc893eaf6e8830cc92eee7af804a8022b7"}
Oct 30 19:20:59.936085 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.935972    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:20:59.937149 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.937123    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-hnr6l" event={"ID":"52a21ddd-696e-4d75-a0e0-424aba32607d","Type":"ContainerStarted","Data":"12622e16fd9c0d2c74fa2b349cdb6876aae30e0dd998a8267ebf47627c4b29df"}
Oct 30 19:20:59.938340 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.938316    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r8gv2" event={"ID":"a6076fa2-677c-4d8e-871c-238ec152066c","Type":"ContainerStarted","Data":"d8b91cef41bee9f5e7d06d94a5ae04d53571b27b73061618d1bcdc3b4e6ad68e"}
Oct 30 19:20:59.978351 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:20:59.978305    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb" podStartSLOduration=0.978293334 podStartE2EDuration="978.293334ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:59.97813319 +0000 UTC m=+13.277395321" watchObservedRunningTime="2026-10-30 19:20:59.978293334 +0000 UTC m=+13.277555513"
Oct 30 19:21:00.943318 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:00.943225    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r8gv2" event={"ID":"a6076fa2-677c-4d8e-871c-238ec152066c","Type":"ContainerStarted","Data":"96208393169d91e95d23b6b4fc05c8b7d25de0de09c91f3a05e35a8594f5802f"}
Oct 30 19:21:00.996423 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:00.996373    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r8gv2" podStartSLOduration=1.946552343 podStartE2EDuration="2.996344675s" totalImagesPullingTime="1.049792332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:00.983756705 +0000 UTC m=+14.283018834" watchObservedRunningTime="2026-10-30 19:21:00.996344675 +0000 UTC m=+14.295606805"
Oct 30 19:21:01.946868 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:01.946832    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r8gv2"
Oct 30 19:21:01.953100 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:01.953037    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-r8gv2"
Oct 30 19:21:02.087439 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:02.087404    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:02.223792 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:02.223714    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:02.888377 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:02.888325    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:02.949521 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:02.949488    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b"]
Oct 30 19:21:02.952662 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:02.952641    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b"
Oct 30 19:21:02.968685 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:02.968662    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 19:21:02.981205 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:02.981181    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:21:02.981205 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:02.981186    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:21:02.983841 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:02.983821    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 19:21:02.984532 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:02.984505    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b"]
Oct 30 19:21:02.993692 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:02.993667    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jhf7r\""
Oct 30 19:21:03.022079 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:03.022050    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:03.058032 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:03.058009    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/993b76e2-e387-4307-9309-96a7073b69cf-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wbc4b\" (UID: \"993b76e2-e387-4307-9309-96a7073b69cf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b"
Oct 30 19:21:03.058156 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:03.058052    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/993b76e2-e387-4307-9309-96a7073b69cf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wbc4b\" (UID: \"993b76e2-e387-4307-9309-96a7073b69cf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b"
Oct 30 19:21:03.058233 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:03.058195    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsczz\" (UniqueName: \"kubernetes.io/projected/993b76e2-e387-4307-9309-96a7073b69cf-kube-api-access-rsczz\") pod \"prometheus-operator-9cd89b74c-wbc4b\" (UID: \"993b76e2-e387-4307-9309-96a7073b69cf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b"
Oct 30 19:21:03.058290 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:03.058267    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/993b76e2-e387-4307-9309-96a7073b69cf-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wbc4b\" (UID: \"993b76e2-e387-4307-9309-96a7073b69cf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b"
Oct 30 19:21:03.069734 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:03.069710    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 19:21:03.159233 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:03.159170    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsczz\" (UniqueName: \"kubernetes.io/projected/993b76e2-e387-4307-9309-96a7073b69cf-kube-api-access-rsczz\") pod \"prometheus-operator-9cd89b74c-wbc4b\" (UID: \"993b76e2-e387-4307-9309-96a7073b69cf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b"
Oct 30 19:21:03.159233 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:03.159227    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/993b76e2-e387-4307-9309-96a7073b69cf-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wbc4b\" (UID: \"993b76e2-e387-4307-9309-96a7073b69cf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b"
Oct 30 19:21:03.159394 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:03.159267    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/993b76e2-e387-4307-9309-96a7073b69cf-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wbc4b\" (UID: \"993b76e2-e387-4307-9309-96a7073b69cf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b"
Oct 30 19:21:03.159394 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:03.159298    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/993b76e2-e387-4307-9309-96a7073b69cf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wbc4b\" (UID: \"993b76e2-e387-4307-9309-96a7073b69cf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b"
Oct 30 19:21:03.160415 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:03.160348    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/993b76e2-e387-4307-9309-96a7073b69cf-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wbc4b\" (UID: \"993b76e2-e387-4307-9309-96a7073b69cf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b"
Oct 30 19:21:03.162099 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:03.162078    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/993b76e2-e387-4307-9309-96a7073b69cf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wbc4b\" (UID: \"993b76e2-e387-4307-9309-96a7073b69cf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b"
Oct 30 19:21:03.162431 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:03.162409    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/993b76e2-e387-4307-9309-96a7073b69cf-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wbc4b\" (UID: \"993b76e2-e387-4307-9309-96a7073b69cf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b"
Oct 30 19:21:03.179214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:03.179188    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsczz\" (UniqueName: \"kubernetes.io/projected/993b76e2-e387-4307-9309-96a7073b69cf-kube-api-access-rsczz\") pod \"prometheus-operator-9cd89b74c-wbc4b\" (UID: \"993b76e2-e387-4307-9309-96a7073b69cf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b"
Oct 30 19:21:03.268440 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:03.268411    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b"
Oct 30 19:21:03.563914 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:03.563762    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b"]
Oct 30 19:21:03.566693 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:21:03.566666    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993b76e2_e387_4307_9309_96a7073b69cf.slice/crio-0134353a9440b445e7da21cf9a57f3f253ff8b527ce643f7789c0990b96c3d40 WatchSource:0}: Error finding container 0134353a9440b445e7da21cf9a57f3f253ff8b527ce643f7789c0990b96c3d40: Status 404 returned error can't find the container with id 0134353a9440b445e7da21cf9a57f3f253ff8b527ce643f7789c0990b96c3d40
Oct 30 19:21:03.807264 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:03.807227    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:03.954904 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:03.954870    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b" event={"ID":"993b76e2-e387-4307-9309-96a7073b69cf","Type":"ContainerStarted","Data":"0134353a9440b445e7da21cf9a57f3f253ff8b527ce643f7789c0990b96c3d40"}
Oct 30 19:21:04.208026 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:04.207953    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:04.353093 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:04.353060    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:04.960799 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:04.960767    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b" event={"ID":"993b76e2-e387-4307-9309-96a7073b69cf","Type":"ContainerStarted","Data":"c51d8904381fe7952c491f6e0f57839acc976aacc32017981dffdbf38dc8db15"}
Oct 30 19:21:05.738897 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:05.738839    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:05.970356 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:05.970317    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b" event={"ID":"993b76e2-e387-4307-9309-96a7073b69cf","Type":"ContainerStarted","Data":"6ffa17f46d458e31b28df937e8e94094185345c3a0a886a53d640404616e76b9"}
Oct 30 19:21:06.010775 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:06.010682    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wbc4b" podStartSLOduration=2.6989946160000002 podStartE2EDuration="4.010665816s" totalImagesPullingTime="1.3116712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:06.010057346 +0000 UTC m=+19.309319474" watchObservedRunningTime="2026-10-30 19:21:06.010665816 +0000 UTC m=+19.309927945"
Oct 30 19:21:06.374589 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:21:06.374552    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:21:06.475506 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:06.475463    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:06.608069 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:06.608035    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:07.159879 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:07.159848    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:07.352234 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:07.352192    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:07.492829 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:07.492753    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:07.647183 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:07.647152    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:08.313652 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.313621    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:08.517083 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.517047    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:08.640782 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.640738    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7tqb7"]
Oct 30 19:21:08.644282 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.644252    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.660983 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.660960    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 19:21:08.660983 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.660975    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 19:21:08.673338 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.673316    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gqsl5\""
Oct 30 19:21:08.691291 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.691269    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 19:21:08.733040 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.733012    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:08.802523 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.802499    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgvv4\" (UniqueName: \"kubernetes.io/projected/5bfa2f12-7480-4a2c-9373-77087aea1a3c-kube-api-access-xgvv4\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.802523 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.802534    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bfa2f12-7480-4a2c-9373-77087aea1a3c-node-exporter-tls\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.802705 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.802570    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bfa2f12-7480-4a2c-9373-77087aea1a3c-root\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.802763 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.802712    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bfa2f12-7480-4a2c-9373-77087aea1a3c-node-exporter-textfile\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.802763 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.802740    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bfa2f12-7480-4a2c-9373-77087aea1a3c-sys\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.802859 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.802764    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bfa2f12-7480-4a2c-9373-77087aea1a3c-metrics-client-ca\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.802859 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.802820    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bfa2f12-7480-4a2c-9373-77087aea1a3c-node-exporter-accelerators-collector-config\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.802948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.802866    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bfa2f12-7480-4a2c-9373-77087aea1a3c-node-exporter-wtmp\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.802948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.802910    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bfa2f12-7480-4a2c-9373-77087aea1a3c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.903927 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.903854    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bfa2f12-7480-4a2c-9373-77087aea1a3c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.903927 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.903906    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgvv4\" (UniqueName: \"kubernetes.io/projected/5bfa2f12-7480-4a2c-9373-77087aea1a3c-kube-api-access-xgvv4\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.904107 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.903934    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bfa2f12-7480-4a2c-9373-77087aea1a3c-node-exporter-tls\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.904107 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.903969    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bfa2f12-7480-4a2c-9373-77087aea1a3c-root\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.904107 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.904005    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bfa2f12-7480-4a2c-9373-77087aea1a3c-node-exporter-textfile\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.904107 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.904040    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bfa2f12-7480-4a2c-9373-77087aea1a3c-sys\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.904107 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.904071    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bfa2f12-7480-4a2c-9373-77087aea1a3c-metrics-client-ca\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.904107 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.904103    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bfa2f12-7480-4a2c-9373-77087aea1a3c-node-exporter-accelerators-collector-config\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.904353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.904130    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bfa2f12-7480-4a2c-9373-77087aea1a3c-node-exporter-wtmp\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.904353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.904292    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bfa2f12-7480-4a2c-9373-77087aea1a3c-node-exporter-wtmp\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.904865 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.904836    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bfa2f12-7480-4a2c-9373-77087aea1a3c-root\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.904865 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.904857    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bfa2f12-7480-4a2c-9373-77087aea1a3c-sys\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.905032 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.904955    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bfa2f12-7480-4a2c-9373-77087aea1a3c-node-exporter-textfile\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.905461 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.905438    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bfa2f12-7480-4a2c-9373-77087aea1a3c-node-exporter-accelerators-collector-config\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.905758 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.905741    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bfa2f12-7480-4a2c-9373-77087aea1a3c-metrics-client-ca\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.912398 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.906992    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bfa2f12-7480-4a2c-9373-77087aea1a3c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.912398 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.907234    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bfa2f12-7480-4a2c-9373-77087aea1a3c-node-exporter-tls\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.916606 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.916583    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:08.925715 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.925695    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgvv4\" (UniqueName: \"kubernetes.io/projected/5bfa2f12-7480-4a2c-9373-77087aea1a3c-kube-api-access-xgvv4\") pod \"node-exporter-7tqb7\" (UID: \"5bfa2f12-7480-4a2c-9373-77087aea1a3c\") " pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.963515 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.963477    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7tqb7"
Oct 30 19:21:08.977549 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:21:08.977521    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfa2f12_7480_4a2c_9373_77087aea1a3c.slice/crio-95d6e037ece9bb7d5fb2dead1ce0a0170c202d36ab6b28d6d9f26e92c991a0c2 WatchSource:0}: Error finding container 95d6e037ece9bb7d5fb2dead1ce0a0170c202d36ab6b28d6d9f26e92c991a0c2: Status 404 returned error can't find the container with id 95d6e037ece9bb7d5fb2dead1ce0a0170c202d36ab6b28d6d9f26e92c991a0c2
Oct 30 19:21:08.983709 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:08.983679    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7tqb7" event={"ID":"5bfa2f12-7480-4a2c-9373-77087aea1a3c","Type":"ContainerStarted","Data":"95d6e037ece9bb7d5fb2dead1ce0a0170c202d36ab6b28d6d9f26e92c991a0c2"}
Oct 30 19:21:09.064545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:09.064515    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:09.217739 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:09.217649    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:09.380068 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:09.380036    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:09.740349 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:09.740315    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:10.119819 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:10.119762    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:10.344530 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:10.344488    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:10.993144 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:10.993104    4712 generic.go:356] "Generic (PLEG): container finished" podID="5bfa2f12-7480-4a2c-9373-77087aea1a3c" containerID="d172e726b51908cecbd5a3144eccb2b2e3e36ad7f2f4f28c45e0c4a01265e520" exitCode=0
Oct 30 19:21:10.993634 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:10.993186    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7tqb7" event={"ID":"5bfa2f12-7480-4a2c-9373-77087aea1a3c","Type":"ContainerDied","Data":"d172e726b51908cecbd5a3144eccb2b2e3e36ad7f2f4f28c45e0c4a01265e520"}
Oct 30 19:21:11.791215 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:11.791178    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:11.936035 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:11.936001    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:11.999425 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:11.999388    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7tqb7" event={"ID":"5bfa2f12-7480-4a2c-9373-77087aea1a3c","Type":"ContainerStarted","Data":"5b325aa197e2b964724aedf91598ad0d2aca24d0ced4418be1cd5283aed905f8"}
Oct 30 19:21:11.999810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:11.999458    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7tqb7" event={"ID":"5bfa2f12-7480-4a2c-9373-77087aea1a3c","Type":"ContainerStarted","Data":"2834758c6d002e2d4b17864c9b3ceeb3216f98caada479047b94be61d55a7bb1"}
Oct 30 19:21:12.068611 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:12.068516    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7tqb7" podStartSLOduration=2.592901344 podStartE2EDuration="4.068501841s" totalImagesPullingTime="1.389111614s" totalInitContainerRuntime="86.488883ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:21:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:12.068085356 +0000 UTC m=+25.367347486" watchObservedRunningTime="2026-10-30 19:21:12.068501841 +0000 UTC m=+25.367763970"
Oct 30 19:21:12.087273 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:12.087233    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:12.938070 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:12.938024    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:13.377103 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:13.377066    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:13.514494 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:13.514431    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:13.651488 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:13.651409    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:14.795870 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:14.795791    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:14.932353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:14.932316    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:15.430491 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:15.430460    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:16.016185 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:16.016155    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-hnr6l" event={"ID":"52a21ddd-696e-4d75-a0e0-424aba32607d","Type":"ContainerStarted","Data":"7e5172f4999fa74c6e155511812ee87ca65825ed49118bceb3baf1d478d9efa2"}
Oct 30 19:21:16.016624 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:16.016221    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-hnr6l"
Oct 30 19:21:16.028413 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:16.025829    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-hnr6l"
Oct 30 19:21:16.080009 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:16.079954    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-hnr6l" podStartSLOduration=2.142967137 podStartE2EDuration="18.079938174s" totalImagesPullingTime="15.936971037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:16.079502012 +0000 UTC m=+29.378764143" watchObservedRunningTime="2026-10-30 19:21:16.079938174 +0000 UTC m=+29.379200304"
Oct 30 19:21:16.557257 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:16.557209    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:16.693736 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:16.693702    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6rkvc"
Oct 30 19:21:16.710325 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:16.710294    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:16.941566 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:16.941527    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:17.086205 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:17.086167    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:17.241807 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:17.241726    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:19.133343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:19.133301    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:19.269467 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:19.269397    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:20.343200 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:20.343129    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:20.485918 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:20.485862    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:20.948542 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:20.948515    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7974c9c7fd-g85kb"
Oct 30 19:21:21.175091 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:21.175052    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:21.304067 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:21.303993    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:21.664209 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:21.664177    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:21.804611 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:21.802291    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:22.200187 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:22.200147    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6dnzx"
Oct 30 19:21:24.140721 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:24.140678    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:24.280491 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:24.280441    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:24.417485 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:24.417414    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:25.625345 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:25.625315    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:25.773944 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:25.773910    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:28.214685 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:28.214652    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:28.357293 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:28.357265    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:29.583437 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:29.583340    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:29.811740 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:29.811711    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:31.582620 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:31.582592    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:31.770216 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:31.770184    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:32.067429 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:32.067403    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:32.220218 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:32.220178    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:36.372875 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:21:36.372834    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:21:38.755095 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:38.755060    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:38.894277 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:38.894238    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:46.824292 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:46.824261    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:46.963218 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:46.963186    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:47.107820 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:47.107750    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:55.875398 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:55.875328    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:56.012441 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:56.012404    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:56.159709 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:56.159636    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:21:56.299466 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:21:56.299432    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:01.219441 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.219409    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-snzrt-w5xjz"]
Oct 30 19:22:01.219831 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.219719    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-snzrt-w5xjz" podUID="9838f46c-794d-4b6f-9f2d-a1a45e21819f" containerName="insights-gathering" containerID="cri-o://99f15990511a6c5f7860ba9ce9fdf6cc29aac7ed1eaf3bb5475d30d41e764b6c" gracePeriod=30
Oct 30 19:22:01.351692 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.351666    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-rrzkg-wdnwr"]
Oct 30 19:22:01.360524 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.360501    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rrzkg-wdnwr"
Oct 30 19:22:01.373240 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.373216    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rrzkg-wdnwr"]
Oct 30 19:22:01.466457 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.466428    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-trusted-ca-bundle\") pod \"periodic-gathering-rrzkg-wdnwr\" (UID: \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\") " pod="openshift-insights/periodic-gathering-rrzkg-wdnwr"
Oct 30 19:22:01.466584 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.466466    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdzn7\" (UniqueName: \"kubernetes.io/projected/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-kube-api-access-xdzn7\") pod \"periodic-gathering-rrzkg-wdnwr\" (UID: \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\") " pod="openshift-insights/periodic-gathering-rrzkg-wdnwr"
Oct 30 19:22:01.466624 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.466585    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-archives-path\") pod \"periodic-gathering-rrzkg-wdnwr\" (UID: \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\") " pod="openshift-insights/periodic-gathering-rrzkg-wdnwr"
Oct 30 19:22:01.466660 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.466634    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-service-ca-bundle\") pod \"periodic-gathering-rrzkg-wdnwr\" (UID: \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\") " pod="openshift-insights/periodic-gathering-rrzkg-wdnwr"
Oct 30 19:22:01.474566 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.474527    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:22:01.567537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.567514    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9838f46c-794d-4b6f-9f2d-a1a45e21819f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9838f46c-794d-4b6f-9f2d-a1a45e21819f-archives-path\") pod \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") "
Oct 30 19:22:01.567663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.567558    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-trusted-ca-bundle\") pod \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") "
Oct 30 19:22:01.567663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.567582    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9838f46c-794d-4b6f-9f2d-a1a45e21819f-kube-api-access-kdmgh\" (UniqueName: \"kubernetes.io/projected/9838f46c-794d-4b6f-9f2d-a1a45e21819f-kube-api-access-kdmgh\") pod \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") "
Oct 30 19:22:01.567663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.567605    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-service-ca-bundle\") pod \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\" (UID: \"9838f46c-794d-4b6f-9f2d-a1a45e21819f\") "
Oct 30 19:22:01.567663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.567662    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-archives-path\") pod \"periodic-gathering-rrzkg-wdnwr\" (UID: \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\") " pod="openshift-insights/periodic-gathering-rrzkg-wdnwr"
Oct 30 19:22:01.567872 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.567685    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-service-ca-bundle\") pod \"periodic-gathering-rrzkg-wdnwr\" (UID: \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\") " pod="openshift-insights/periodic-gathering-rrzkg-wdnwr"
Oct 30 19:22:01.567872 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.567728    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-trusted-ca-bundle\") pod \"periodic-gathering-rrzkg-wdnwr\" (UID: \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\") " pod="openshift-insights/periodic-gathering-rrzkg-wdnwr"
Oct 30 19:22:01.567872 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.567748    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzn7\" (UniqueName: \"kubernetes.io/projected/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-kube-api-access-xdzn7\") pod \"periodic-gathering-rrzkg-wdnwr\" (UID: \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\") " pod="openshift-insights/periodic-gathering-rrzkg-wdnwr"
Oct 30 19:22:01.568030 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.567886    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9838f46c-794d-4b6f-9f2d-a1a45e21819f-archives-path" pod "9838f46c-794d-4b6f-9f2d-a1a45e21819f" (UID: "9838f46c-794d-4b6f-9f2d-a1a45e21819f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:22:01.568030 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.567940    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-trusted-ca-bundle" pod "9838f46c-794d-4b6f-9f2d-a1a45e21819f" (UID: "9838f46c-794d-4b6f-9f2d-a1a45e21819f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:01.568030 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.568000    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-service-ca-bundle" pod "9838f46c-794d-4b6f-9f2d-a1a45e21819f" (UID: "9838f46c-794d-4b6f-9f2d-a1a45e21819f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:22:01.568178 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.568070    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-archives-path\") pod \"periodic-gathering-rrzkg-wdnwr\" (UID: \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\") " pod="openshift-insights/periodic-gathering-rrzkg-wdnwr"
Oct 30 19:22:01.568437 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.568356    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-service-ca-bundle\") pod \"periodic-gathering-rrzkg-wdnwr\" (UID: \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\") " pod="openshift-insights/periodic-gathering-rrzkg-wdnwr"
Oct 30 19:22:01.568546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.568526    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-trusted-ca-bundle\") pod \"periodic-gathering-rrzkg-wdnwr\" (UID: \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\") " pod="openshift-insights/periodic-gathering-rrzkg-wdnwr"
Oct 30 19:22:01.569874 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.569853    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9838f46c-794d-4b6f-9f2d-a1a45e21819f-kube-api-access-kdmgh" pod "9838f46c-794d-4b6f-9f2d-a1a45e21819f" (UID: "9838f46c-794d-4b6f-9f2d-a1a45e21819f"). InnerVolumeSpecName "kube-api-access-kdmgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:22:01.587678 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.587651    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdzn7\" (UniqueName: \"kubernetes.io/projected/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-kube-api-access-xdzn7\") pod \"periodic-gathering-rrzkg-wdnwr\" (UID: \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\") " pod="openshift-insights/periodic-gathering-rrzkg-wdnwr"
Oct 30 19:22:01.668237 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.668217    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdmgh\" (UniqueName: \"kubernetes.io/projected/9838f46c-794d-4b6f-9f2d-a1a45e21819f-kube-api-access-kdmgh\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:22:01.668237 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.668237    4712 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-service-ca-bundle\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:22:01.668354 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.668247    4712 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9838f46c-794d-4b6f-9f2d-a1a45e21819f-archives-path\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:22:01.668354 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.668256    4712 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9838f46c-794d-4b6f-9f2d-a1a45e21819f-trusted-ca-bundle\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:22:01.683695 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:01.683678    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rrzkg-wdnwr"
Oct 30 19:22:02.005143 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:02.005098    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rrzkg-wdnwr"]
Oct 30 19:22:02.007115 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:22:02.007090    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda456a78b_272b_4a7f_bb0f_56c6c6ae0408.slice/crio-73849c5d7bf984360504e91b03b4ff58d1ad73c9f1dc55834ad578a8e641f67a WatchSource:0}: Error finding container 73849c5d7bf984360504e91b03b4ff58d1ad73c9f1dc55834ad578a8e641f67a: Status 404 returned error can't find the container with id 73849c5d7bf984360504e91b03b4ff58d1ad73c9f1dc55834ad578a8e641f67a
Oct 30 19:22:02.166699 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:02.166660    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rrzkg-wdnwr" event={"ID":"a456a78b-272b-4a7f-bb0f-56c6c6ae0408","Type":"ContainerStarted","Data":"e7b9cc3ef3f9ece48d6d2cea75111c21b5ce3566d7c93019e9d4348afa92fa81"}
Oct 30 19:22:02.166845 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:02.166717    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rrzkg-wdnwr" event={"ID":"a456a78b-272b-4a7f-bb0f-56c6c6ae0408","Type":"ContainerStarted","Data":"73849c5d7bf984360504e91b03b4ff58d1ad73c9f1dc55834ad578a8e641f67a"}
Oct 30 19:22:02.167783 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:02.167759    4712 generic.go:356] "Generic (PLEG): container finished" podID="9838f46c-794d-4b6f-9f2d-a1a45e21819f" containerID="99f15990511a6c5f7860ba9ce9fdf6cc29aac7ed1eaf3bb5475d30d41e764b6c" exitCode=2
Oct 30 19:22:02.167914 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:02.167817    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-snzrt-w5xjz"
Oct 30 19:22:02.167914 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:02.167829    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-snzrt-w5xjz" event={"ID":"9838f46c-794d-4b6f-9f2d-a1a45e21819f","Type":"ContainerDied","Data":"99f15990511a6c5f7860ba9ce9fdf6cc29aac7ed1eaf3bb5475d30d41e764b6c"}
Oct 30 19:22:02.167914 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:02.167873    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-snzrt-w5xjz" event={"ID":"9838f46c-794d-4b6f-9f2d-a1a45e21819f","Type":"ContainerDied","Data":"b1fdf1c7824e8af0757de524fb1b5161ca4c06bd4777aadccf3d7c78419cbd60"}
Oct 30 19:22:02.167914 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:02.167904    4712 scope.go:122] "RemoveContainer" containerID="99f15990511a6c5f7860ba9ce9fdf6cc29aac7ed1eaf3bb5475d30d41e764b6c"
Oct 30 19:22:02.176708 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:02.176686    4712 scope.go:122] "RemoveContainer" containerID="99f15990511a6c5f7860ba9ce9fdf6cc29aac7ed1eaf3bb5475d30d41e764b6c"
Oct 30 19:22:02.176976 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:22:02.176956    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f15990511a6c5f7860ba9ce9fdf6cc29aac7ed1eaf3bb5475d30d41e764b6c\": container with ID starting with 99f15990511a6c5f7860ba9ce9fdf6cc29aac7ed1eaf3bb5475d30d41e764b6c not found: ID does not exist" containerID="99f15990511a6c5f7860ba9ce9fdf6cc29aac7ed1eaf3bb5475d30d41e764b6c"
Oct 30 19:22:02.177023 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:02.176985    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f15990511a6c5f7860ba9ce9fdf6cc29aac7ed1eaf3bb5475d30d41e764b6c"} err="failed to get container status \"99f15990511a6c5f7860ba9ce9fdf6cc29aac7ed1eaf3bb5475d30d41e764b6c\": rpc error: code = NotFound desc = could not find container \"99f15990511a6c5f7860ba9ce9fdf6cc29aac7ed1eaf3bb5475d30d41e764b6c\": container with ID starting with 99f15990511a6c5f7860ba9ce9fdf6cc29aac7ed1eaf3bb5475d30d41e764b6c not found: ID does not exist"
Oct 30 19:22:02.207545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:02.207502    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-rrzkg-wdnwr" podStartSLOduration=1.207492541 podStartE2EDuration="1.207492541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:02.206790304 +0000 UTC m=+75.506052433" watchObservedRunningTime="2026-10-30 19:22:02.207492541 +0000 UTC m=+75.506754648"
Oct 30 19:22:02.241723 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:02.241693    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-snzrt-w5xjz"]
Oct 30 19:22:02.248163 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:02.248144    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-snzrt-w5xjz"]
Oct 30 19:22:02.848994 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:02.848952    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9838f46c-794d-4b6f-9f2d-a1a45e21819f" path="/var/lib/kubelet/pods/9838f46c-794d-4b6f-9f2d-a1a45e21819f/volumes"
Oct 30 19:22:06.371914 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:22:06.371883    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:22:12.645518 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:12.645478    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:12.908346 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:12.908267    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:18.641676 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:18.641628    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:18.849522 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:18.849486    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:25.524196 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:25.524160    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:25.693943 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:25.693909    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:28.590010 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:28.589976    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:29.675437 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:29.675399    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:30.697204 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:30.697175    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:30.859647 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:30.859603    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:31.083543 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:31.083454    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:22:36.371472 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:22:36.371436    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:22:52.580446 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:22:52.580410    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:06.372709 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:23:06.372634    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:23:26.426294 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:26.426216    4712 generic.go:356] "Generic (PLEG): container finished" podID="a456a78b-272b-4a7f-bb0f-56c6c6ae0408" containerID="e7b9cc3ef3f9ece48d6d2cea75111c21b5ce3566d7c93019e9d4348afa92fa81" exitCode=0
Oct 30 19:23:26.426294 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:26.426254    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rrzkg-wdnwr" event={"ID":"a456a78b-272b-4a7f-bb0f-56c6c6ae0408","Type":"ContainerDied","Data":"e7b9cc3ef3f9ece48d6d2cea75111c21b5ce3566d7c93019e9d4348afa92fa81"}
Oct 30 19:23:27.543662 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:27.543642    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rrzkg-wdnwr"
Oct 30 19:23:27.622671 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:27.622649    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-archives-path\") pod \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\" (UID: \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\") "
Oct 30 19:23:27.622793 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:27.622701    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-trusted-ca-bundle\") pod \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\" (UID: \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\") "
Oct 30 19:23:27.622793 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:27.622723    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-service-ca-bundle\") pod \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\" (UID: \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\") "
Oct 30 19:23:27.622793 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:27.622755    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-kube-api-access-xdzn7\" (UniqueName: \"kubernetes.io/projected/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-kube-api-access-xdzn7\") pod \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\" (UID: \"a456a78b-272b-4a7f-bb0f-56c6c6ae0408\") "
Oct 30 19:23:27.622985 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:27.622963    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-archives-path" pod "a456a78b-272b-4a7f-bb0f-56c6c6ae0408" (UID: "a456a78b-272b-4a7f-bb0f-56c6c6ae0408"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:27.623049 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:27.622997    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-trusted-ca-bundle" pod "a456a78b-272b-4a7f-bb0f-56c6c6ae0408" (UID: "a456a78b-272b-4a7f-bb0f-56c6c6ae0408"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:27.623148 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:27.623126    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-service-ca-bundle" pod "a456a78b-272b-4a7f-bb0f-56c6c6ae0408" (UID: "a456a78b-272b-4a7f-bb0f-56c6c6ae0408"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:27.624699 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:27.624678    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-kube-api-access-xdzn7" pod "a456a78b-272b-4a7f-bb0f-56c6c6ae0408" (UID: "a456a78b-272b-4a7f-bb0f-56c6c6ae0408"). InnerVolumeSpecName "kube-api-access-xdzn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:27.724129 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:27.724083    4712 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-archives-path\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:23:27.724129 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:27.724103    4712 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-trusted-ca-bundle\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:23:27.724129 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:27.724112    4712 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-service-ca-bundle\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:23:27.724129 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:27.724120    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdzn7\" (UniqueName: \"kubernetes.io/projected/a456a78b-272b-4a7f-bb0f-56c6c6ae0408-kube-api-access-xdzn7\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:23:28.434292 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:28.434260    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rrzkg-wdnwr" event={"ID":"a456a78b-272b-4a7f-bb0f-56c6c6ae0408","Type":"ContainerDied","Data":"73849c5d7bf984360504e91b03b4ff58d1ad73c9f1dc55834ad578a8e641f67a"}
Oct 30 19:23:28.434292 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:28.434284    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rrzkg-wdnwr"
Oct 30 19:23:28.434486 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:28.434306    4712 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73849c5d7bf984360504e91b03b4ff58d1ad73c9f1dc55834ad578a8e641f67a"
Oct 30 19:23:29.011947 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:29.011916    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:29.153549 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:29.153520    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:29.773236 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:23:29.773205    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:23:36.371606 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:23:36.371571    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:24:01.447346 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:01.447316    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:01.589338 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:01.589310    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:06.371984 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:24:06.371947    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:24:29.914571 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:29.914490    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:30.329309 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:30.329232    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:30.563969 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:30.563932    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:30.814185 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:30.814156    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:32.724041 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.724006    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"]
Oct 30 19:24:32.724643 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.724356    4712 cpu_manager.go:414] "containerMap: removing container" podUID="a456a78b-272b-4a7f-bb0f-56c6c6ae0408" containerName="insights-gathering"
Oct 30 19:24:32.724643 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.724387    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a456a78b-272b-4a7f-bb0f-56c6c6ae0408" containerName="insights-gathering"
Oct 30 19:24:32.724643 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.724401    4712 cpu_manager.go:414] "containerMap: removing container" podUID="9838f46c-794d-4b6f-9f2d-a1a45e21819f" containerName="insights-gathering"
Oct 30 19:24:32.724643 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.724410    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9838f46c-794d-4b6f-9f2d-a1a45e21819f" containerName="insights-gathering"
Oct 30 19:24:32.724643 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.724498    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="a456a78b-272b-4a7f-bb0f-56c6c6ae0408" containerName="insights-gathering"
Oct 30 19:24:32.724643 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.724510    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="9838f46c-794d-4b6f-9f2d-a1a45e21819f" containerName="insights-gathering"
Oct 30 19:24:32.726209 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.726188    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.745725 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.745704    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:24:32.745880 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.745868    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:24:32.757822 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.757800    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 19:24:32.757928 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.757800    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 19:24:32.757928 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.757841    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6vscc\""
Oct 30 19:24:32.757928 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.757857    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 19:24:32.758136 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.758112    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:24:32.760240 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.760222    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"]
Oct 30 19:24:32.829796 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.829770    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.829881 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.829800    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.829881 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.829824    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.829881 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.829844    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.829986 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.829901    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.830022 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.830003    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.830057 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.830038    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd4g4\" (UniqueName: \"kubernetes.io/projected/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-kube-api-access-gd4g4\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.857651 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.857626    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:32.930486 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.930465    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.930568 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.930502    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4g4\" (UniqueName: \"kubernetes.io/projected/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-kube-api-access-gd4g4\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.930708 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.930693    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.930759 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.930730    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.930798 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.930760    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.930866 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.930849    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.930909 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.930883    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.931170 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.931151    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.933016 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.932996    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.933095 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.933016    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.933095 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.933031    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.933167 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.933118    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.952203 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.952182    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:32.952565 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:32.952548    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd4g4\" (UniqueName: \"kubernetes.io/projected/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-kube-api-access-gd4g4\") pod \"istiod-openshift-gateway-7876bfcb74-p9wp4\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:33.034407 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:33.034351    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:33.035293 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:33.035275    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:33.337675 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:33.337648    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"]
Oct 30 19:24:33.339251 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:24:33.339229    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d5c6ed_f9f4_4408_80c8_3c0ee7742ef1.slice/crio-6334f3cf4287adb5dbcfcbfb20d02d8aeef49676173f4368f81744475dbb56a7 WatchSource:0}: Error finding container 6334f3cf4287adb5dbcfcbfb20d02d8aeef49676173f4368f81744475dbb56a7: Status 404 returned error can't find the container with id 6334f3cf4287adb5dbcfcbfb20d02d8aeef49676173f4368f81744475dbb56a7
Oct 30 19:24:33.638457 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:33.638426    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4" event={"ID":"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1","Type":"ContainerStarted","Data":"6334f3cf4287adb5dbcfcbfb20d02d8aeef49676173f4368f81744475dbb56a7"}
Oct 30 19:24:34.010944 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:34.010872    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:36.085303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:36.085271    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:24:36.085612 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:36.085340    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:24:36.371574 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:24:36.371540    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:24:36.652279 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:36.652212    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4" event={"ID":"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1","Type":"ContainerStarted","Data":"e316ed59c749f31a2b9f8007017764152c5a35834aea9bc0e0588589bf6bbb16"}
Oct 30 19:24:36.652425 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:36.652315    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:36.706069 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:36.706007    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4" podStartSLOduration=1.962528751 podStartE2EDuration="4.705988197s" totalImagesPullingTime="2.743459446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:36.705162537 +0000 UTC m=+230.004424677" watchObservedRunningTime="2026-10-30 19:24:36.705988197 +0000 UTC m=+230.005250329"
Oct 30 19:24:36.791506 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:36.791474    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:36.950175 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:36.950108    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:37.551535 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:37.551501    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:37.657302 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:37.657266    4712 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-p9wp4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:24:37.657484 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:37.657335    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4" podUID="59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:24:37.719446 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:37.719409    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:40.657285 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:40.657255    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:24:40.799333 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:40.799308    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:57.241736 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:57.241687    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:24:57.376562 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:24:57.376532    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:06.371898 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:25:06.371863    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:25:29.076938 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:29.076902    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:31.309054 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:31.309020    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:31.649412 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:31.649382    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:31.795120 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:31.795088    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:32.300725 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.300689    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:32.537871 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.537835    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"]
Oct 30 19:25:32.541107 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.541055    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.583005 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.582948    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:32.618028 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.618004    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"]
Oct 30 19:25:32.620373 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.620342    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc7521d6-8864-41b1-a0e7-5e661b811a58-tmp\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-v9mb6\" (UID: \"dc7521d6-8864-41b1-a0e7-5e661b811a58\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.620482 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.620391    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc7521d6-8864-41b1-a0e7-5e661b811a58-metrics-cert\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-v9mb6\" (UID: \"dc7521d6-8864-41b1-a0e7-5e661b811a58\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.620482 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.620420    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc7521d6-8864-41b1-a0e7-5e661b811a58-manifests\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-v9mb6\" (UID: \"dc7521d6-8864-41b1-a0e7-5e661b811a58\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.620482 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.620443    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc7521d6-8864-41b1-a0e7-5e661b811a58-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-v9mb6\" (UID: \"dc7521d6-8864-41b1-a0e7-5e661b811a58\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.620584 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.620569    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc7521d6-8864-41b1-a0e7-5e661b811a58-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-v9mb6\" (UID: \"dc7521d6-8864-41b1-a0e7-5e661b811a58\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.620616 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.620603    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdx6l\" (UniqueName: \"kubernetes.io/projected/dc7521d6-8864-41b1-a0e7-5e661b811a58-kube-api-access-gdx6l\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-v9mb6\" (UID: \"dc7521d6-8864-41b1-a0e7-5e661b811a58\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.621851 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.621833    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 19:25:32.622011 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.621997    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 19:25:32.622420 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.622395    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-8hq8l\""
Oct 30 19:25:32.622539 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.622459    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 19:25:32.622940 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.622924    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 19:25:32.622988 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.622967    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 19:25:32.721217 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.721189    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc7521d6-8864-41b1-a0e7-5e661b811a58-tmp\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-v9mb6\" (UID: \"dc7521d6-8864-41b1-a0e7-5e661b811a58\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.721217 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.721217    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc7521d6-8864-41b1-a0e7-5e661b811a58-metrics-cert\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-v9mb6\" (UID: \"dc7521d6-8864-41b1-a0e7-5e661b811a58\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.721428 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.721235    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc7521d6-8864-41b1-a0e7-5e661b811a58-manifests\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-v9mb6\" (UID: \"dc7521d6-8864-41b1-a0e7-5e661b811a58\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.721428 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.721257    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc7521d6-8864-41b1-a0e7-5e661b811a58-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-v9mb6\" (UID: \"dc7521d6-8864-41b1-a0e7-5e661b811a58\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.721428 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.721320    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc7521d6-8864-41b1-a0e7-5e661b811a58-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-v9mb6\" (UID: \"dc7521d6-8864-41b1-a0e7-5e661b811a58\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.721428 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.721386    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdx6l\" (UniqueName: \"kubernetes.io/projected/dc7521d6-8864-41b1-a0e7-5e661b811a58-kube-api-access-gdx6l\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-v9mb6\" (UID: \"dc7521d6-8864-41b1-a0e7-5e661b811a58\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.721680 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.721656    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc7521d6-8864-41b1-a0e7-5e661b811a58-manifests\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-v9mb6\" (UID: \"dc7521d6-8864-41b1-a0e7-5e661b811a58\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.723679 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.723656    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc7521d6-8864-41b1-a0e7-5e661b811a58-tmp\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-v9mb6\" (UID: \"dc7521d6-8864-41b1-a0e7-5e661b811a58\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.723818 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.723798    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc7521d6-8864-41b1-a0e7-5e661b811a58-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-v9mb6\" (UID: \"dc7521d6-8864-41b1-a0e7-5e661b811a58\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.723900 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.723858    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc7521d6-8864-41b1-a0e7-5e661b811a58-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-v9mb6\" (UID: \"dc7521d6-8864-41b1-a0e7-5e661b811a58\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.723982 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.723963    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc7521d6-8864-41b1-a0e7-5e661b811a58-metrics-cert\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-v9mb6\" (UID: \"dc7521d6-8864-41b1-a0e7-5e661b811a58\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.743424 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.743397    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdx6l\" (UniqueName: \"kubernetes.io/projected/dc7521d6-8864-41b1-a0e7-5e661b811a58-kube-api-access-gdx6l\") pod \"opendatahub-operator-controller-manager-5d58dddd7d-v9mb6\" (UID: \"dc7521d6-8864-41b1-a0e7-5e661b811a58\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.855059 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.855007    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:32.897136 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:32.897110    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:33.055937 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:33.055905    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:33.241427 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:33.241405    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"]
Oct 30 19:25:33.244166 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:25:33.244142    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7521d6_8864_41b1_a0e7_5e661b811a58.slice/crio-9f6bdf0fbba7f482adc79637cfac65490a471ded4b02618d207f63ee91f7ccee WatchSource:0}: Error finding container 9f6bdf0fbba7f482adc79637cfac65490a471ded4b02618d207f63ee91f7ccee: Status 404 returned error can't find the container with id 9f6bdf0fbba7f482adc79637cfac65490a471ded4b02618d207f63ee91f7ccee
Oct 30 19:25:33.246142 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:33.246116    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:33.842706 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:33.842667    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6" event={"ID":"dc7521d6-8864-41b1-a0e7-5e661b811a58","Type":"ContainerStarted","Data":"9f6bdf0fbba7f482adc79637cfac65490a471ded4b02618d207f63ee91f7ccee"}
Oct 30 19:25:35.060791 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:35.060756    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:35.207462 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:35.207422    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:36.371347 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:25:36.371314    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:25:36.853526 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:36.853464    4712 generic.go:356] "Generic (PLEG): container finished" podID="dc7521d6-8864-41b1-a0e7-5e661b811a58" containerID="9a22938bc75c902dd66d03e0b1e4eb4a140523681350cce8f147c6cdfb288163" exitCode=0
Oct 30 19:25:36.853526 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:36.853513    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6" event={"ID":"dc7521d6-8864-41b1-a0e7-5e661b811a58","Type":"ContainerDied","Data":"9a22938bc75c902dd66d03e0b1e4eb4a140523681350cce8f147c6cdfb288163"}
Oct 30 19:25:37.625371 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:37.625326    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:37.860201 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:37.860158    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6" event={"ID":"dc7521d6-8864-41b1-a0e7-5e661b811a58","Type":"ContainerStarted","Data":"2f8018875c6c190f3ad7d00c22c162b5f6f425cf27b8f2c999f67037dae312a9"}
Oct 30 19:25:37.860402 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:37.860269    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:37.962627 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:37.962528    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6" podStartSLOduration=2.38594725 podStartE2EDuration="5.962514084s" totalImagesPullingTime="3.15741143s" totalInitContainerRuntime="419.155404ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:37.950213961 +0000 UTC m=+291.249476125" watchObservedRunningTime="2026-10-30 19:25:37.962514084 +0000 UTC m=+291.261776212"
Oct 30 19:25:40.345982 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:40.345943    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:40.514385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:40.511727    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:42.860557 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:42.860528    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:43.028242 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:43.028213    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:44.020742 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:44.020711    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:44.177142 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:44.177110    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:44.322354 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:44.322284    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:45.878622 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:45.878584    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:46.036772 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:46.036743    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:46.861619 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:46.861593    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:25:46.862112 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:46.862091    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:25:48.340177 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:48.340141    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:48.490506 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:48.490468    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:48.865256 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:48.865232    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5d58dddd7d-v9mb6"
Oct 30 19:25:50.227890 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:50.227858    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:54.856583 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:54.856546    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:56.107387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:56.107334    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:57.282157 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:57.282130    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:57.474765 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:57.474736    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:57.700304 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:57.700272    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:58.424732 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:58.424701    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:59.085143 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:59.085107    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:25:59.402311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:25:59.402279    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:00.559144 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:00.559106    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:00.744751 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:00.744722    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:03.636941 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:03.636900    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:04.033050 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:04.032972    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:05.867656 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:05.867624    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:06.372688 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:26:06.372655    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:26:07.448417 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:07.448384    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:07.598810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:07.598777    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:07.985787 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:07.985759    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:08.291322 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.291259    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:08.481664 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.481631    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:08.747315 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.747287    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58487bf9b4-lck4n"]
Oct 30 19:26:08.750673 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.750646    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58487bf9b4-lck4n"
Oct 30 19:26:08.765627 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.765580    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:08.773692 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.773622    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58487bf9b4-lck4n"]
Oct 30 19:26:08.827470 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.827437    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 19:26:08.827628 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.827489    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9h9vk\""
Oct 30 19:26:08.827628 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.827552    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 19:26:08.874504 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.874475    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqkcw\" (UniqueName: \"kubernetes.io/projected/cd1c6ba9-e503-4755-89c2-1700e80c4e12-kube-api-access-sqkcw\") pod \"kube-auth-proxy-58487bf9b4-lck4n\" (UID: \"cd1c6ba9-e503-4755-89c2-1700e80c4e12\") " pod="openshift-ingress/kube-auth-proxy-58487bf9b4-lck4n"
Oct 30 19:26:08.874637 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.874516    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1c6ba9-e503-4755-89c2-1700e80c4e12-tls-certs\") pod \"kube-auth-proxy-58487bf9b4-lck4n\" (UID: \"cd1c6ba9-e503-4755-89c2-1700e80c4e12\") " pod="openshift-ingress/kube-auth-proxy-58487bf9b4-lck4n"
Oct 30 19:26:08.874637 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.874546    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6ba9-e503-4755-89c2-1700e80c4e12-tmp\") pod \"kube-auth-proxy-58487bf9b4-lck4n\" (UID: \"cd1c6ba9-e503-4755-89c2-1700e80c4e12\") " pod="openshift-ingress/kube-auth-proxy-58487bf9b4-lck4n"
Oct 30 19:26:08.939423 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.939388    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:08.975617 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.975587    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6ba9-e503-4755-89c2-1700e80c4e12-tmp\") pod \"kube-auth-proxy-58487bf9b4-lck4n\" (UID: \"cd1c6ba9-e503-4755-89c2-1700e80c4e12\") " pod="openshift-ingress/kube-auth-proxy-58487bf9b4-lck4n"
Oct 30 19:26:08.975749 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.975650    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqkcw\" (UniqueName: \"kubernetes.io/projected/cd1c6ba9-e503-4755-89c2-1700e80c4e12-kube-api-access-sqkcw\") pod \"kube-auth-proxy-58487bf9b4-lck4n\" (UID: \"cd1c6ba9-e503-4755-89c2-1700e80c4e12\") " pod="openshift-ingress/kube-auth-proxy-58487bf9b4-lck4n"
Oct 30 19:26:08.975749 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.975694    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1c6ba9-e503-4755-89c2-1700e80c4e12-tls-certs\") pod \"kube-auth-proxy-58487bf9b4-lck4n\" (UID: \"cd1c6ba9-e503-4755-89c2-1700e80c4e12\") " pod="openshift-ingress/kube-auth-proxy-58487bf9b4-lck4n"
Oct 30 19:26:08.977761 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.977733    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd1c6ba9-e503-4755-89c2-1700e80c4e12-tmp\") pod \"kube-auth-proxy-58487bf9b4-lck4n\" (UID: \"cd1c6ba9-e503-4755-89c2-1700e80c4e12\") " pod="openshift-ingress/kube-auth-proxy-58487bf9b4-lck4n"
Oct 30 19:26:08.977992 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.977974    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1c6ba9-e503-4755-89c2-1700e80c4e12-tls-certs\") pod \"kube-auth-proxy-58487bf9b4-lck4n\" (UID: \"cd1c6ba9-e503-4755-89c2-1700e80c4e12\") " pod="openshift-ingress/kube-auth-proxy-58487bf9b4-lck4n"
Oct 30 19:26:08.995075 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:08.995047    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqkcw\" (UniqueName: \"kubernetes.io/projected/cd1c6ba9-e503-4755-89c2-1700e80c4e12-kube-api-access-sqkcw\") pod \"kube-auth-proxy-58487bf9b4-lck4n\" (UID: \"cd1c6ba9-e503-4755-89c2-1700e80c4e12\") " pod="openshift-ingress/kube-auth-proxy-58487bf9b4-lck4n"
Oct 30 19:26:09.069579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:09.069523    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58487bf9b4-lck4n"
Oct 30 19:26:09.100307 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:09.100281    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:09.322167 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:09.322100    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:09.376540 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:09.376512    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58487bf9b4-lck4n"]
Oct 30 19:26:09.377798 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:26:09.377766    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1c6ba9_e503_4755_89c2_1700e80c4e12.slice/crio-2557e3a37e19e46f1815a99fe8fb92b925518f271b97c00c0ba7373e916e14e9 WatchSource:0}: Error finding container 2557e3a37e19e46f1815a99fe8fb92b925518f271b97c00c0ba7373e916e14e9: Status 404 returned error can't find the container with id 2557e3a37e19e46f1815a99fe8fb92b925518f271b97c00c0ba7373e916e14e9
Oct 30 19:26:09.379739 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:09.379718    4712 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:26:09.507261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:09.507229    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:09.796942 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:09.796910    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:09.963158 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:09.963121    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58487bf9b4-lck4n" event={"ID":"cd1c6ba9-e503-4755-89c2-1700e80c4e12","Type":"ContainerStarted","Data":"2557e3a37e19e46f1815a99fe8fb92b925518f271b97c00c0ba7373e916e14e9"}
Oct 30 19:26:09.963828 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:09.963758    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:10.487348 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:10.487309    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:10.758741 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:10.758664    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:10.926927 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:10.926850    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:11.269686 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.269651    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:11.335414 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.335383    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgcgf"]
Oct 30 19:26:11.338949 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.338922    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgcgf"
Oct 30 19:26:11.366967 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.366928    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nrqr9\""
Oct 30 19:26:11.366967 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.366948    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 19:26:11.367314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.367262    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 19:26:11.367573 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.367516    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 19:26:11.367647 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.367597    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 19:26:11.377849 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.377826    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgcgf"]
Oct 30 19:26:11.421317 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.421283    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:11.495751 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.495713    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvlmd\" (UniqueName: \"kubernetes.io/projected/f055912e-ab5f-4098-ba65-bc664b7aee30-kube-api-access-fvlmd\") pod \"kuadrant-console-plugin-66c4fb575d-fgcgf\" (UID: \"f055912e-ab5f-4098-ba65-bc664b7aee30\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgcgf"
Oct 30 19:26:11.495918 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.495777    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f055912e-ab5f-4098-ba65-bc664b7aee30-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-fgcgf\" (UID: \"f055912e-ab5f-4098-ba65-bc664b7aee30\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgcgf"
Oct 30 19:26:11.495918 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.495889    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f055912e-ab5f-4098-ba65-bc664b7aee30-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-fgcgf\" (UID: \"f055912e-ab5f-4098-ba65-bc664b7aee30\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgcgf"
Oct 30 19:26:11.581392 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.581285    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:11.599740 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.599707    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f055912e-ab5f-4098-ba65-bc664b7aee30-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-fgcgf\" (UID: \"f055912e-ab5f-4098-ba65-bc664b7aee30\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgcgf"
Oct 30 19:26:11.599920 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.599794    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvlmd\" (UniqueName: \"kubernetes.io/projected/f055912e-ab5f-4098-ba65-bc664b7aee30-kube-api-access-fvlmd\") pod \"kuadrant-console-plugin-66c4fb575d-fgcgf\" (UID: \"f055912e-ab5f-4098-ba65-bc664b7aee30\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgcgf"
Oct 30 19:26:11.599920 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.599838    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f055912e-ab5f-4098-ba65-bc664b7aee30-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-fgcgf\" (UID: \"f055912e-ab5f-4098-ba65-bc664b7aee30\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgcgf"
Oct 30 19:26:11.600694 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.600664    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f055912e-ab5f-4098-ba65-bc664b7aee30-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-fgcgf\" (UID: \"f055912e-ab5f-4098-ba65-bc664b7aee30\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgcgf"
Oct 30 19:26:11.605104 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.605057    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f055912e-ab5f-4098-ba65-bc664b7aee30-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-fgcgf\" (UID: \"f055912e-ab5f-4098-ba65-bc664b7aee30\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgcgf"
Oct 30 19:26:11.624239 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.624210    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvlmd\" (UniqueName: \"kubernetes.io/projected/f055912e-ab5f-4098-ba65-bc664b7aee30-kube-api-access-fvlmd\") pod \"kuadrant-console-plugin-66c4fb575d-fgcgf\" (UID: \"f055912e-ab5f-4098-ba65-bc664b7aee30\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgcgf"
Oct 30 19:26:11.665089 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.665059    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgcgf"
Oct 30 19:26:11.791580 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.791558    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:11.972608 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.972579    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58487bf9b4-lck4n" event={"ID":"cd1c6ba9-e503-4755-89c2-1700e80c4e12","Type":"ContainerStarted","Data":"02d589268abc69a5e44268c1c1103c07a21bf67c8b5ba997ec900b31edeab9fa"}
Oct 30 19:26:11.973739 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:11.973720    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgcgf"]
Oct 30 19:26:11.974888 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:26:11.974866    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf055912e_ab5f_4098_ba65_bc664b7aee30.slice/crio-7c871dcb2687b8789db5f415fbd5b5386efe584efca07370d8d424a36a68b3f7 WatchSource:0}: Error finding container 7c871dcb2687b8789db5f415fbd5b5386efe584efca07370d8d424a36a68b3f7: Status 404 returned error can't find the container with id 7c871dcb2687b8789db5f415fbd5b5386efe584efca07370d8d424a36a68b3f7
Oct 30 19:26:12.019663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:12.019618    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58487bf9b4-lck4n" podStartSLOduration=1.6705723190000001 podStartE2EDuration="4.019605207s" totalImagesPullingTime="2.349032888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:12.018704681 +0000 UTC m=+325.317966809" watchObservedRunningTime="2026-10-30 19:26:12.019605207 +0000 UTC m=+325.318867336"
Oct 30 19:26:12.162756 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:12.162718    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:12.311565 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:12.311484    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:12.977842 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:12.977799    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgcgf" event={"ID":"f055912e-ab5f-4098-ba65-bc664b7aee30","Type":"ContainerStarted","Data":"7c871dcb2687b8789db5f415fbd5b5386efe584efca07370d8d424a36a68b3f7"}
Oct 30 19:26:14.872188 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:14.872152    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:15.047306 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.047269    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:15.380849 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.380819    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:15.473802 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.473766    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"]
Oct 30 19:26:15.479688 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.479666    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"
Oct 30 19:26:15.511258 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.511107    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-qrlbk\""
Oct 30 19:26:15.511258 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.511110    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:26:15.511258 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.511151    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 19:26:15.511258 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.511152    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 19:26:15.511643 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.511344    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 19:26:15.511795 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.511762    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 19:26:15.512392 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.512306    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"]
Oct 30 19:26:15.528898 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.528865    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:15.632208 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.632126    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps4v5\" (UniqueName: \"kubernetes.io/projected/ed2d4b0d-05a0-4069-a24a-f1206c8f75c7-kube-api-access-ps4v5\") pod \"jobset-controller-manager-74864577cf-nw6p4\" (UID: \"ed2d4b0d-05a0-4069-a24a-f1206c8f75c7\") " pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"
Oct 30 19:26:15.632208 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.632200    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ed2d4b0d-05a0-4069-a24a-f1206c8f75c7-manager-config\") pod \"jobset-controller-manager-74864577cf-nw6p4\" (UID: \"ed2d4b0d-05a0-4069-a24a-f1206c8f75c7\") " pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"
Oct 30 19:26:15.632453 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.632232    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed2d4b0d-05a0-4069-a24a-f1206c8f75c7-cert\") pod \"jobset-controller-manager-74864577cf-nw6p4\" (UID: \"ed2d4b0d-05a0-4069-a24a-f1206c8f75c7\") " pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"
Oct 30 19:26:15.632453 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.632286    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed2d4b0d-05a0-4069-a24a-f1206c8f75c7-metrics-certs\") pod \"jobset-controller-manager-74864577cf-nw6p4\" (UID: \"ed2d4b0d-05a0-4069-a24a-f1206c8f75c7\") " pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"
Oct 30 19:26:15.668972 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.668934    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:15.733050 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.733019    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps4v5\" (UniqueName: \"kubernetes.io/projected/ed2d4b0d-05a0-4069-a24a-f1206c8f75c7-kube-api-access-ps4v5\") pod \"jobset-controller-manager-74864577cf-nw6p4\" (UID: \"ed2d4b0d-05a0-4069-a24a-f1206c8f75c7\") " pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"
Oct 30 19:26:15.733204 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.733065    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ed2d4b0d-05a0-4069-a24a-f1206c8f75c7-manager-config\") pod \"jobset-controller-manager-74864577cf-nw6p4\" (UID: \"ed2d4b0d-05a0-4069-a24a-f1206c8f75c7\") " pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"
Oct 30 19:26:15.733204 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.733091    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed2d4b0d-05a0-4069-a24a-f1206c8f75c7-cert\") pod \"jobset-controller-manager-74864577cf-nw6p4\" (UID: \"ed2d4b0d-05a0-4069-a24a-f1206c8f75c7\") " pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"
Oct 30 19:26:15.733204 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.733124    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed2d4b0d-05a0-4069-a24a-f1206c8f75c7-metrics-certs\") pod \"jobset-controller-manager-74864577cf-nw6p4\" (UID: \"ed2d4b0d-05a0-4069-a24a-f1206c8f75c7\") " pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"
Oct 30 19:26:15.733834 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.733806    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ed2d4b0d-05a0-4069-a24a-f1206c8f75c7-manager-config\") pod \"jobset-controller-manager-74864577cf-nw6p4\" (UID: \"ed2d4b0d-05a0-4069-a24a-f1206c8f75c7\") " pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"
Oct 30 19:26:15.735647 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.735623    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed2d4b0d-05a0-4069-a24a-f1206c8f75c7-metrics-certs\") pod \"jobset-controller-manager-74864577cf-nw6p4\" (UID: \"ed2d4b0d-05a0-4069-a24a-f1206c8f75c7\") " pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"
Oct 30 19:26:15.735768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.735652    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed2d4b0d-05a0-4069-a24a-f1206c8f75c7-cert\") pod \"jobset-controller-manager-74864577cf-nw6p4\" (UID: \"ed2d4b0d-05a0-4069-a24a-f1206c8f75c7\") " pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"
Oct 30 19:26:15.752954 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.752924    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps4v5\" (UniqueName: \"kubernetes.io/projected/ed2d4b0d-05a0-4069-a24a-f1206c8f75c7-kube-api-access-ps4v5\") pod \"jobset-controller-manager-74864577cf-nw6p4\" (UID: \"ed2d4b0d-05a0-4069-a24a-f1206c8f75c7\") " pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"
Oct 30 19:26:15.796112 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:15.796075    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"
Oct 30 19:26:16.103530 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:16.103499    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"]
Oct 30 19:26:16.105286 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:26:16.105244    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2d4b0d_05a0_4069_a24a_f1206c8f75c7.slice/crio-8699341a75923b893b9e58991a649c630a0d9ef74c8edb8e71456b155e644f4d WatchSource:0}: Error finding container 8699341a75923b893b9e58991a649c630a0d9ef74c8edb8e71456b155e644f4d: Status 404 returned error can't find the container with id 8699341a75923b893b9e58991a649c630a0d9ef74c8edb8e71456b155e644f4d
Oct 30 19:26:16.995286 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:16.995247    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4" event={"ID":"ed2d4b0d-05a0-4069-a24a-f1206c8f75c7","Type":"ContainerStarted","Data":"8699341a75923b893b9e58991a649c630a0d9ef74c8edb8e71456b155e644f4d"}
Oct 30 19:26:16.997201 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:16.997167    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgcgf" event={"ID":"f055912e-ab5f-4098-ba65-bc664b7aee30","Type":"ContainerStarted","Data":"1ef6a89993b8070159a00fcba1116caae054c4a8dc6b9c1eab18378ccac29ad6"}
Oct 30 19:26:17.039325 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:17.039270    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-fgcgf" podStartSLOduration=1.683176477 podStartE2EDuration="6.03925438s" totalImagesPullingTime="4.356077903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:17.037637876 +0000 UTC m=+330.336900004" watchObservedRunningTime="2026-10-30 19:26:17.03925438 +0000 UTC m=+330.338516511"
Oct 30 19:26:19.006642 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:19.006604    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4" event={"ID":"ed2d4b0d-05a0-4069-a24a-f1206c8f75c7","Type":"ContainerStarted","Data":"3cdc2e13c1300b72240d0a99366c5829c35037f6fe04239430ba98ead3f04cfc"}
Oct 30 19:26:19.007055 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:19.006706    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"
Oct 30 19:26:19.048855 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:19.048811    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4" podStartSLOduration=1.801634701 podStartE2EDuration="4.048801066s" totalImagesPullingTime="2.247166365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:19.048146194 +0000 UTC m=+332.347408324" watchObservedRunningTime="2026-10-30 19:26:19.048801066 +0000 UTC m=+332.348063194"
Oct 30 19:26:23.916272 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:23.916239    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:24.070306 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:24.070260    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:26.518340 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:26.518302    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:26.668235 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:26.668207    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:29.710435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:29.710401    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:29.849991 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:29.849959    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:30.016137 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:30.016062    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-74864577cf-nw6p4"
Oct 30 19:26:30.106699 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:30.106664    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:30.567454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:30.567423    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:30.710106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:30.710081    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:31.346114 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:31.346074    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:34.313135 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:34.313097    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:34.453985 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:34.453939    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:34.706782 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:34.706754    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:34.848527 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:34.848492    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:34.993020 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:34.992951    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:36.372886 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:26:36.372852    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:26:42.916191 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:42.916152    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:43.075406 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:43.075351    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:50.730866 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:50.730834    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:50.888217 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:50.888183    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:51.024459 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:51.024388    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:54.584051 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:54.584015    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:54.728958 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:54.728926    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:56.909217 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:56.909189    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:26:57.052142 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:26:57.052111    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:06.372312 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:27:06.372268    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:27:09.789664 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:09.789627    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:09.950824 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:09.950792    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:10.113811 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:10.113743    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:10.270548 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:10.270500    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:10.531535 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:10.531504    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:12.364157 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:12.364125    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:12.504614 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:12.504577    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:12.651109 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:12.651059    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:12.794338 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:12.794302    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:19.983835 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:19.983803    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"]
Oct 30 19:27:19.984272 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:19.984010    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4" podUID="59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1" containerName="discovery" containerID="cri-o://e316ed59c749f31a2b9f8007017764152c5a35834aea9bc0e0588589bf6bbb16" gracePeriod=30
Oct 30 19:27:20.223636 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.223613    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:27:20.228542 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.228519    4712 generic.go:356] "Generic (PLEG): container finished" podID="59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1" containerID="e316ed59c749f31a2b9f8007017764152c5a35834aea9bc0e0588589bf6bbb16" exitCode=0
Oct 30 19:27:20.228622 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.228554    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"
Oct 30 19:27:20.228622 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.228597    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4" event={"ID":"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1","Type":"ContainerDied","Data":"e316ed59c749f31a2b9f8007017764152c5a35834aea9bc0e0588589bf6bbb16"}
Oct 30 19:27:20.228696 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.228648    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4" event={"ID":"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1","Type":"ContainerDied","Data":"6334f3cf4287adb5dbcfcbfb20d02d8aeef49676173f4368f81744475dbb56a7"}
Oct 30 19:27:20.228696 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.228675    4712 scope.go:122] "RemoveContainer" containerID="e316ed59c749f31a2b9f8007017764152c5a35834aea9bc0e0588589bf6bbb16"
Oct 30 19:27:20.235971 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.235953    4712 scope.go:122] "RemoveContainer" containerID="e316ed59c749f31a2b9f8007017764152c5a35834aea9bc0e0588589bf6bbb16"
Oct 30 19:27:20.236234 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:27:20.236216    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e316ed59c749f31a2b9f8007017764152c5a35834aea9bc0e0588589bf6bbb16\": container with ID starting with e316ed59c749f31a2b9f8007017764152c5a35834aea9bc0e0588589bf6bbb16 not found: ID does not exist" containerID="e316ed59c749f31a2b9f8007017764152c5a35834aea9bc0e0588589bf6bbb16"
Oct 30 19:27:20.236280 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.236243    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e316ed59c749f31a2b9f8007017764152c5a35834aea9bc0e0588589bf6bbb16"} err="failed to get container status \"e316ed59c749f31a2b9f8007017764152c5a35834aea9bc0e0588589bf6bbb16\": rpc error: code = NotFound desc = could not find container \"e316ed59c749f31a2b9f8007017764152c5a35834aea9bc0e0588589bf6bbb16\": container with ID starting with e316ed59c749f31a2b9f8007017764152c5a35834aea9bc0e0588589bf6bbb16 not found: ID does not exist"
Oct 30 19:27:20.294394 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.294345    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-kubeconfig\") pod \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") "
Oct 30 19:27:20.294520 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.294415    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-csr-dns-cert\") pod \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") "
Oct 30 19:27:20.294520 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.294443    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-csr-ca-configmap\") pod \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") "
Oct 30 19:27:20.294520 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.294484    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-local-certs\") pod \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") "
Oct 30 19:27:20.294688 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.294520    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-kube-api-access-gd4g4\" (UniqueName: \"kubernetes.io/projected/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-kube-api-access-gd4g4\") pod \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") "
Oct 30 19:27:20.294688 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.294554    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-cacerts\" (UniqueName: \"kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-cacerts\") pod \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") "
Oct 30 19:27:20.294688 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.294601    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-token\" (UniqueName: \"kubernetes.io/projected/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-token\") pod \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\" (UID: \"59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1\") "
Oct 30 19:27:20.294942 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.294906    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-csr-ca-configmap" pod "59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1" (UID: "59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:20.296845 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.296787    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-csr-dns-cert" pod "59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1" (UID: "59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:20.296932 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.296868    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-cacerts" pod "59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1" (UID: "59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:20.297074 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.297052    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-kubeconfig" pod "59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1" (UID: "59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:20.297162 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.297100    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-local-certs" pod "59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1" (UID: "59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:27:20.297206 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.297177    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-token" pod "59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1" (UID: "59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:20.297241 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.297198    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-kube-api-access-gd4g4" pod "59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1" (UID: "59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1"). InnerVolumeSpecName "kube-api-access-gd4g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:20.395160 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.395140    4712 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-cacerts\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:27:20.395160 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.395158    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:27:20.395275 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.395169    4712 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-kubeconfig\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:27:20.395275 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.395178    4712 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-csr-dns-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:27:20.395275 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.395187    4712 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-istio-csr-ca-configmap\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:27:20.395275 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.395196    4712 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-local-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:27:20.395275 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.395204    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd4g4\" (UniqueName: \"kubernetes.io/projected/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1-kube-api-access-gd4g4\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:27:20.571943 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.571919    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"]
Oct 30 19:27:20.584514 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.584486    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-p9wp4"]
Oct 30 19:27:20.850445 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:20.850388    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1" path="/var/lib/kubelet/pods/59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1/volumes"
Oct 30 19:27:26.322395 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:26.322343    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:26.459411 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:26.459378    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:27.779896 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:27.779861    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:27.984342 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:27.984312    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:28.001128 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.001098    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"]
Oct 30 19:27:28.001424 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.001397    4712 cpu_manager.go:414] "containerMap: removing container" podUID="59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1" containerName="discovery"
Oct 30 19:27:28.001424 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.001426    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1" containerName="discovery"
Oct 30 19:27:28.001510 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.001484    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="59d5c6ed-f9f4-4408-80c8-3c0ee7742ef1" containerName="discovery"
Oct 30 19:27:28.005266 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.005248    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"
Oct 30 19:27:28.033766 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.033721    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:27:28.033766 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.033761    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-g97rg\""
Oct 30 19:27:28.034116 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.034101    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 19:27:28.034200 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.034159    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:27:28.034253 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.034205    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:27:28.046453 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.046426    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5wxm\" (UniqueName: \"kubernetes.io/projected/78594ef4-736b-4687-baa5-fe9e1d3f0920-kube-api-access-t5wxm\") pod \"maas-api-6d669c7fcd-fmsrp\" (UID: \"78594ef4-736b-4687-baa5-fe9e1d3f0920\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"
Oct 30 19:27:28.046564 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.046460    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78594ef4-736b-4687-baa5-fe9e1d3f0920-maas-api-tls\") pod \"maas-api-6d669c7fcd-fmsrp\" (UID: \"78594ef4-736b-4687-baa5-fe9e1d3f0920\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"
Oct 30 19:27:28.046564 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.046482    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78594ef4-736b-4687-baa5-fe9e1d3f0920-metrics-certs\") pod \"maas-api-6d669c7fcd-fmsrp\" (UID: \"78594ef4-736b-4687-baa5-fe9e1d3f0920\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"
Oct 30 19:27:28.046670 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.046647    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78594ef4-736b-4687-baa5-fe9e1d3f0920-openshift-service-ca\") pod \"maas-api-6d669c7fcd-fmsrp\" (UID: \"78594ef4-736b-4687-baa5-fe9e1d3f0920\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"
Oct 30 19:27:28.047373 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.047341    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"]
Oct 30 19:27:28.147679 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.147654    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78594ef4-736b-4687-baa5-fe9e1d3f0920-openshift-service-ca\") pod \"maas-api-6d669c7fcd-fmsrp\" (UID: \"78594ef4-736b-4687-baa5-fe9e1d3f0920\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"
Oct 30 19:27:28.147791 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.147691    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5wxm\" (UniqueName: \"kubernetes.io/projected/78594ef4-736b-4687-baa5-fe9e1d3f0920-kube-api-access-t5wxm\") pod \"maas-api-6d669c7fcd-fmsrp\" (UID: \"78594ef4-736b-4687-baa5-fe9e1d3f0920\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"
Oct 30 19:27:28.147791 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.147710    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78594ef4-736b-4687-baa5-fe9e1d3f0920-maas-api-tls\") pod \"maas-api-6d669c7fcd-fmsrp\" (UID: \"78594ef4-736b-4687-baa5-fe9e1d3f0920\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"
Oct 30 19:27:28.147791 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.147728    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78594ef4-736b-4687-baa5-fe9e1d3f0920-metrics-certs\") pod \"maas-api-6d669c7fcd-fmsrp\" (UID: \"78594ef4-736b-4687-baa5-fe9e1d3f0920\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"
Oct 30 19:27:28.148296 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.148275    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78594ef4-736b-4687-baa5-fe9e1d3f0920-openshift-service-ca\") pod \"maas-api-6d669c7fcd-fmsrp\" (UID: \"78594ef4-736b-4687-baa5-fe9e1d3f0920\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"
Oct 30 19:27:28.150179 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.150153    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78594ef4-736b-4687-baa5-fe9e1d3f0920-metrics-certs\") pod \"maas-api-6d669c7fcd-fmsrp\" (UID: \"78594ef4-736b-4687-baa5-fe9e1d3f0920\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"
Oct 30 19:27:28.150293 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.150182    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78594ef4-736b-4687-baa5-fe9e1d3f0920-maas-api-tls\") pod \"maas-api-6d669c7fcd-fmsrp\" (UID: \"78594ef4-736b-4687-baa5-fe9e1d3f0920\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"
Oct 30 19:27:28.168570 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.168548    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:28.168696 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.168674    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5wxm\" (UniqueName: \"kubernetes.io/projected/78594ef4-736b-4687-baa5-fe9e1d3f0920-kube-api-access-t5wxm\") pod \"maas-api-6d669c7fcd-fmsrp\" (UID: \"78594ef4-736b-4687-baa5-fe9e1d3f0920\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"
Oct 30 19:27:28.317134 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.317071    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"
Oct 30 19:27:28.610006 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:28.609977    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"]
Oct 30 19:27:28.611380 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:27:28.611339    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78594ef4_736b_4687_baa5_fe9e1d3f0920.slice/crio-502adfdd77471878cc90706c167b5f5f8fcb43f9acd2b6ee7bd7d95bc698ee8c WatchSource:0}: Error finding container 502adfdd77471878cc90706c167b5f5f8fcb43f9acd2b6ee7bd7d95bc698ee8c: Status 404 returned error can't find the container with id 502adfdd77471878cc90706c167b5f5f8fcb43f9acd2b6ee7bd7d95bc698ee8c
Oct 30 19:27:29.270140 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:29.270096    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp" event={"ID":"78594ef4-736b-4687-baa5-fe9e1d3f0920","Type":"ContainerStarted","Data":"502adfdd77471878cc90706c167b5f5f8fcb43f9acd2b6ee7bd7d95bc698ee8c"}
Oct 30 19:27:30.363473 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:30.363443    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:31.279459 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:31.279391    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp" event={"ID":"78594ef4-736b-4687-baa5-fe9e1d3f0920","Type":"ContainerStarted","Data":"da361bbb1e5e5323eddffe594ed339ba76e904f9777bb78db26db69a9d64e22b"}
Oct 30 19:27:31.279596 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:31.279531    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"
Oct 30 19:27:31.320270 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:31.320225    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp" podStartSLOduration=1.95434318 podStartE2EDuration="4.320201219s" totalImagesPullingTime="2.365858039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:31.318827886 +0000 UTC m=+404.618090015" watchObservedRunningTime="2026-10-30 19:27:31.320201219 +0000 UTC m=+404.619463347"
Oct 30 19:27:36.372293 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:27:36.372259    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:27:37.288064 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:37.288036    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"
Oct 30 19:27:37.849861 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:37.849810    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:38.275218 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:38.275184    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:27:38.386712 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:38.386682    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"]
Oct 30 19:27:38.389558 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:38.389542    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"
Oct 30 19:27:38.422093 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:38.422070    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"]
Oct 30 19:27:38.518847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:38.518812    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcsrk\" (UniqueName: \"kubernetes.io/projected/a92f3725-c731-4b4c-a272-61442b9ee11c-kube-api-access-rcsrk\") pod \"maas-api-5f5c657c85-pl4k4\" (UID: \"a92f3725-c731-4b4c-a272-61442b9ee11c\") " pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"
Oct 30 19:27:38.518999 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:38.518859    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92f3725-c731-4b4c-a272-61442b9ee11c-openshift-service-ca\") pod \"maas-api-5f5c657c85-pl4k4\" (UID: \"a92f3725-c731-4b4c-a272-61442b9ee11c\") " pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"
Oct 30 19:27:38.518999 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:38.518876    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92f3725-c731-4b4c-a272-61442b9ee11c-metrics-certs\") pod \"maas-api-5f5c657c85-pl4k4\" (UID: \"a92f3725-c731-4b4c-a272-61442b9ee11c\") " pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"
Oct 30 19:27:38.519070 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:38.518996    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a92f3725-c731-4b4c-a272-61442b9ee11c-maas-api-tls\") pod \"maas-api-5f5c657c85-pl4k4\" (UID: \"a92f3725-c731-4b4c-a272-61442b9ee11c\") " pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"
Oct 30 19:27:38.619714 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:38.619686    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a92f3725-c731-4b4c-a272-61442b9ee11c-maas-api-tls\") pod \"maas-api-5f5c657c85-pl4k4\" (UID: \"a92f3725-c731-4b4c-a272-61442b9ee11c\") " pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"
Oct 30 19:27:38.619853 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:38.619740    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcsrk\" (UniqueName: \"kubernetes.io/projected/a92f3725-c731-4b4c-a272-61442b9ee11c-kube-api-access-rcsrk\") pod \"maas-api-5f5c657c85-pl4k4\" (UID: \"a92f3725-c731-4b4c-a272-61442b9ee11c\") " pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"
Oct 30 19:27:38.619853 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:38.619765    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92f3725-c731-4b4c-a272-61442b9ee11c-openshift-service-ca\") pod \"maas-api-5f5c657c85-pl4k4\" (UID: \"a92f3725-c731-4b4c-a272-61442b9ee11c\") " pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"
Oct 30 19:27:38.619853 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:38.619780    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92f3725-c731-4b4c-a272-61442b9ee11c-metrics-certs\") pod \"maas-api-5f5c657c85-pl4k4\" (UID: \"a92f3725-c731-4b4c-a272-61442b9ee11c\") " pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"
Oct 30 19:27:38.620431 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:38.620407    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92f3725-c731-4b4c-a272-61442b9ee11c-openshift-service-ca\") pod \"maas-api-5f5c657c85-pl4k4\" (UID: \"a92f3725-c731-4b4c-a272-61442b9ee11c\") " pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"
Oct 30 19:27:38.622100 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:38.622070    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92f3725-c731-4b4c-a272-61442b9ee11c-metrics-certs\") pod \"maas-api-5f5c657c85-pl4k4\" (UID: \"a92f3725-c731-4b4c-a272-61442b9ee11c\") " pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"
Oct 30 19:27:38.622187 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:38.622160    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a92f3725-c731-4b4c-a272-61442b9ee11c-maas-api-tls\") pod \"maas-api-5f5c657c85-pl4k4\" (UID: \"a92f3725-c731-4b4c-a272-61442b9ee11c\") " pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"
Oct 30 19:27:38.642193 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:38.642171    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcsrk\" (UniqueName: \"kubernetes.io/projected/a92f3725-c731-4b4c-a272-61442b9ee11c-kube-api-access-rcsrk\") pod \"maas-api-5f5c657c85-pl4k4\" (UID: \"a92f3725-c731-4b4c-a272-61442b9ee11c\") " pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"
Oct 30 19:27:38.701306 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:38.701282    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"
Oct 30 19:27:39.014753 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:39.014723    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"]
Oct 30 19:27:39.016403 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:27:39.016376    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92f3725_c731_4b4c_a272_61442b9ee11c.slice/crio-2bcde6ddf9101a5f59ccfb3c12eb99e9eff4bf37fcd9f4bbf7daacd1918c2104 WatchSource:0}: Error finding container 2bcde6ddf9101a5f59ccfb3c12eb99e9eff4bf37fcd9f4bbf7daacd1918c2104: Status 404 returned error can't find the container with id 2bcde6ddf9101a5f59ccfb3c12eb99e9eff4bf37fcd9f4bbf7daacd1918c2104
Oct 30 19:27:39.310306 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:39.310242    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4" event={"ID":"a92f3725-c731-4b4c-a272-61442b9ee11c","Type":"ContainerStarted","Data":"2bcde6ddf9101a5f59ccfb3c12eb99e9eff4bf37fcd9f4bbf7daacd1918c2104"}
Oct 30 19:27:40.315626 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:40.315594    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4" event={"ID":"a92f3725-c731-4b4c-a272-61442b9ee11c","Type":"ContainerStarted","Data":"7c542a958105b40c2f694b20e47ce218cc38b0809bbcd8cd125c21c62651d601"}
Oct 30 19:27:40.315993 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:40.315715    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"
Oct 30 19:27:40.358849 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:40.358801    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4" podStartSLOduration=1.988686372 podStartE2EDuration="2.358786223s" totalImagesPullingTime="370.099851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:40.35786388 +0000 UTC m=+413.657126008" watchObservedRunningTime="2026-10-30 19:27:40.358786223 +0000 UTC m=+413.658048343"
Oct 30 19:27:46.325251 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:46.325220    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"
Oct 30 19:27:46.400795 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:46.400758    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"]
Oct 30 19:27:46.400998 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:46.400974    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp" podUID="78594ef4-736b-4687-baa5-fe9e1d3f0920" containerName="maas-api" containerID="cri-o://da361bbb1e5e5323eddffe594ed339ba76e904f9777bb78db26db69a9d64e22b" gracePeriod=30
Oct 30 19:27:46.634619 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:46.634595    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"
Oct 30 19:27:46.779900 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:46.779874    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78594ef4-736b-4687-baa5-fe9e1d3f0920-metrics-certs\" (UniqueName: \"kubernetes.io/secret/78594ef4-736b-4687-baa5-fe9e1d3f0920-metrics-certs\") pod \"78594ef4-736b-4687-baa5-fe9e1d3f0920\" (UID: \"78594ef4-736b-4687-baa5-fe9e1d3f0920\") "
Oct 30 19:27:46.780032 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:46.779906    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78594ef4-736b-4687-baa5-fe9e1d3f0920-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78594ef4-736b-4687-baa5-fe9e1d3f0920-openshift-service-ca\") pod \"78594ef4-736b-4687-baa5-fe9e1d3f0920\" (UID: \"78594ef4-736b-4687-baa5-fe9e1d3f0920\") "
Oct 30 19:27:46.780032 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:46.779983    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78594ef4-736b-4687-baa5-fe9e1d3f0920-kube-api-access-t5wxm\" (UniqueName: \"kubernetes.io/projected/78594ef4-736b-4687-baa5-fe9e1d3f0920-kube-api-access-t5wxm\") pod \"78594ef4-736b-4687-baa5-fe9e1d3f0920\" (UID: \"78594ef4-736b-4687-baa5-fe9e1d3f0920\") "
Oct 30 19:27:46.780032 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:46.780002    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78594ef4-736b-4687-baa5-fe9e1d3f0920-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78594ef4-736b-4687-baa5-fe9e1d3f0920-maas-api-tls\") pod \"78594ef4-736b-4687-baa5-fe9e1d3f0920\" (UID: \"78594ef4-736b-4687-baa5-fe9e1d3f0920\") "
Oct 30 19:27:46.780350 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:46.780321    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78594ef4-736b-4687-baa5-fe9e1d3f0920-openshift-service-ca" pod "78594ef4-736b-4687-baa5-fe9e1d3f0920" (UID: "78594ef4-736b-4687-baa5-fe9e1d3f0920"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:27:46.781975 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:46.781953    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78594ef4-736b-4687-baa5-fe9e1d3f0920-metrics-certs" pod "78594ef4-736b-4687-baa5-fe9e1d3f0920" (UID: "78594ef4-736b-4687-baa5-fe9e1d3f0920"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:46.782062 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:46.782024    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78594ef4-736b-4687-baa5-fe9e1d3f0920-kube-api-access-t5wxm" pod "78594ef4-736b-4687-baa5-fe9e1d3f0920" (UID: "78594ef4-736b-4687-baa5-fe9e1d3f0920"). InnerVolumeSpecName "kube-api-access-t5wxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:46.782133 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:46.782113    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78594ef4-736b-4687-baa5-fe9e1d3f0920-maas-api-tls" pod "78594ef4-736b-4687-baa5-fe9e1d3f0920" (UID: "78594ef4-736b-4687-baa5-fe9e1d3f0920"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:46.881437 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:46.881386    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5wxm\" (UniqueName: \"kubernetes.io/projected/78594ef4-736b-4687-baa5-fe9e1d3f0920-kube-api-access-t5wxm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:27:46.881437 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:46.881411    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78594ef4-736b-4687-baa5-fe9e1d3f0920-maas-api-tls\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:27:46.881437 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:46.881424    4712 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78594ef4-736b-4687-baa5-fe9e1d3f0920-metrics-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:27:46.881437 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:46.881436    4712 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78594ef4-736b-4687-baa5-fe9e1d3f0920-openshift-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:27:47.342556 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:47.342477    4712 generic.go:356] "Generic (PLEG): container finished" podID="78594ef4-736b-4687-baa5-fe9e1d3f0920" containerID="da361bbb1e5e5323eddffe594ed339ba76e904f9777bb78db26db69a9d64e22b" exitCode=0
Oct 30 19:27:47.342556 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:47.342527    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"
Oct 30 19:27:47.343011 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:47.342555    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp" event={"ID":"78594ef4-736b-4687-baa5-fe9e1d3f0920","Type":"ContainerDied","Data":"da361bbb1e5e5323eddffe594ed339ba76e904f9777bb78db26db69a9d64e22b"}
Oct 30 19:27:47.343011 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:47.342607    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp" event={"ID":"78594ef4-736b-4687-baa5-fe9e1d3f0920","Type":"ContainerDied","Data":"502adfdd77471878cc90706c167b5f5f8fcb43f9acd2b6ee7bd7d95bc698ee8c"}
Oct 30 19:27:47.343011 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:47.342629    4712 scope.go:122] "RemoveContainer" containerID="da361bbb1e5e5323eddffe594ed339ba76e904f9777bb78db26db69a9d64e22b"
Oct 30 19:27:47.350357 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:47.350342    4712 scope.go:122] "RemoveContainer" containerID="da361bbb1e5e5323eddffe594ed339ba76e904f9777bb78db26db69a9d64e22b"
Oct 30 19:27:47.350619 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:27:47.350598    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da361bbb1e5e5323eddffe594ed339ba76e904f9777bb78db26db69a9d64e22b\": container with ID starting with da361bbb1e5e5323eddffe594ed339ba76e904f9777bb78db26db69a9d64e22b not found: ID does not exist" containerID="da361bbb1e5e5323eddffe594ed339ba76e904f9777bb78db26db69a9d64e22b"
Oct 30 19:27:47.350692 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:47.350632    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da361bbb1e5e5323eddffe594ed339ba76e904f9777bb78db26db69a9d64e22b"} err="failed to get container status \"da361bbb1e5e5323eddffe594ed339ba76e904f9777bb78db26db69a9d64e22b\": rpc error: code = NotFound desc = could not find container \"da361bbb1e5e5323eddffe594ed339ba76e904f9777bb78db26db69a9d64e22b\": container with ID starting with da361bbb1e5e5323eddffe594ed339ba76e904f9777bb78db26db69a9d64e22b not found: ID does not exist"
Oct 30 19:27:47.395419 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:47.395398    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"]
Oct 30 19:27:47.408555 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:47.408526    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-fmsrp"]
Oct 30 19:27:48.849680 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:27:48.849650    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78594ef4-736b-4687-baa5-fe9e1d3f0920" path="/var/lib/kubelet/pods/78594ef4-736b-4687-baa5-fe9e1d3f0920/volumes"
Oct 30 19:28:00.664848 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:28:00.664808    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:06.372835 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:28:06.372803    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:28:30.353326 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:28:30.353290    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:36.372509 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:28:36.372475    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:28:56.261224 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:28:56.261193    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:28:56.516158 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:28:56.516087    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:06.372613 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:29:06.372525    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:29:18.612528 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.612495    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"]
Oct 30 19:29:18.612974 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.612760    4712 cpu_manager.go:414] "containerMap: removing container" podUID="78594ef4-736b-4687-baa5-fe9e1d3f0920" containerName="maas-api"
Oct 30 19:29:18.612974 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.612770    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78594ef4-736b-4687-baa5-fe9e1d3f0920" containerName="maas-api"
Oct 30 19:29:18.612974 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.612816    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="78594ef4-736b-4687-baa5-fe9e1d3f0920" containerName="maas-api"
Oct 30 19:29:18.615623 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.615596    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.630195 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.630173    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 19:29:18.642058 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.642034    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 19:29:18.642225 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.642204    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 19:29:18.642299 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.642248    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tlm72\""
Oct 30 19:29:18.643448 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.643429    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"]
Oct 30 19:29:18.691570 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.691543    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06773fac-ac83-4d67-902f-da073082dd63-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xzskw\" (UID: \"06773fac-ac83-4d67-902f-da073082dd63\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.691570 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.691571    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06773fac-ac83-4d67-902f-da073082dd63-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xzskw\" (UID: \"06773fac-ac83-4d67-902f-da073082dd63\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.691713 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.691593    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06773fac-ac83-4d67-902f-da073082dd63-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xzskw\" (UID: \"06773fac-ac83-4d67-902f-da073082dd63\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.691713 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.691669    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06773fac-ac83-4d67-902f-da073082dd63-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xzskw\" (UID: \"06773fac-ac83-4d67-902f-da073082dd63\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.691713 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.691705    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06773fac-ac83-4d67-902f-da073082dd63-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xzskw\" (UID: \"06773fac-ac83-4d67-902f-da073082dd63\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.691825 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.691802    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79knr\" (UniqueName: \"kubernetes.io/projected/06773fac-ac83-4d67-902f-da073082dd63-kube-api-access-79knr\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xzskw\" (UID: \"06773fac-ac83-4d67-902f-da073082dd63\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.725502 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.725470    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:18.792637 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.792612    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79knr\" (UniqueName: \"kubernetes.io/projected/06773fac-ac83-4d67-902f-da073082dd63-kube-api-access-79knr\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xzskw\" (UID: \"06773fac-ac83-4d67-902f-da073082dd63\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.792792 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.792664    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06773fac-ac83-4d67-902f-da073082dd63-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xzskw\" (UID: \"06773fac-ac83-4d67-902f-da073082dd63\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.792792 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.792693    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06773fac-ac83-4d67-902f-da073082dd63-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xzskw\" (UID: \"06773fac-ac83-4d67-902f-da073082dd63\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.792792 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.792724    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06773fac-ac83-4d67-902f-da073082dd63-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xzskw\" (UID: \"06773fac-ac83-4d67-902f-da073082dd63\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.792792 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.792763    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06773fac-ac83-4d67-902f-da073082dd63-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xzskw\" (UID: \"06773fac-ac83-4d67-902f-da073082dd63\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.792961 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.792792    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06773fac-ac83-4d67-902f-da073082dd63-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xzskw\" (UID: \"06773fac-ac83-4d67-902f-da073082dd63\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.793157 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.793134    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06773fac-ac83-4d67-902f-da073082dd63-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xzskw\" (UID: \"06773fac-ac83-4d67-902f-da073082dd63\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.793217 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.793180    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06773fac-ac83-4d67-902f-da073082dd63-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xzskw\" (UID: \"06773fac-ac83-4d67-902f-da073082dd63\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.793217 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.793207    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06773fac-ac83-4d67-902f-da073082dd63-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xzskw\" (UID: \"06773fac-ac83-4d67-902f-da073082dd63\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.794860 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.794839    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06773fac-ac83-4d67-902f-da073082dd63-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xzskw\" (UID: \"06773fac-ac83-4d67-902f-da073082dd63\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.795219 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.795198    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06773fac-ac83-4d67-902f-da073082dd63-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xzskw\" (UID: \"06773fac-ac83-4d67-902f-da073082dd63\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.811904 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.811882    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79knr\" (UniqueName: \"kubernetes.io/projected/06773fac-ac83-4d67-902f-da073082dd63-kube-api-access-79knr\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xzskw\" (UID: \"06773fac-ac83-4d67-902f-da073082dd63\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:18.931032 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:18.931005    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:19.221077 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:19.221054    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"]
Oct 30 19:29:19.222530 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:29:19.222497    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06773fac_ac83_4d67_902f_da073082dd63.slice/crio-8236d589047e6fe63f1ff40a6e1fc4fddcccd71855060652408c6721fc84aa70 WatchSource:0}: Error finding container 8236d589047e6fe63f1ff40a6e1fc4fddcccd71855060652408c6721fc84aa70: Status 404 returned error can't find the container with id 8236d589047e6fe63f1ff40a6e1fc4fddcccd71855060652408c6721fc84aa70
Oct 30 19:29:19.680637 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:19.680560    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw" event={"ID":"06773fac-ac83-4d67-902f-da073082dd63","Type":"ContainerStarted","Data":"8236d589047e6fe63f1ff40a6e1fc4fddcccd71855060652408c6721fc84aa70"}
Oct 30 19:29:20.686001 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:20.685968    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw" event={"ID":"06773fac-ac83-4d67-902f-da073082dd63","Type":"ContainerStarted","Data":"4db64e26da6ea7f4fc3d3e94c607ace4b69058a009337352f06d2370f724f0d4"}
Oct 30 19:29:20.727481 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:20.727437    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw" podStartSLOduration=1.68914885 podStartE2EDuration="2.727424586s" totalImagesPullingTime="1.038275736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:20.726230863 +0000 UTC m=+514.025492994" watchObservedRunningTime="2026-10-30 19:29:20.727424586 +0000 UTC m=+514.026686752"
Oct 30 19:29:20.931953 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:20.931922    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:20.938339 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:20.938280    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"]
Oct 30 19:29:20.941554 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:20.941536    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:20.960779 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:20.960760    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 19:29:20.961766 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:20.961749    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"]
Oct 30 19:29:21.010486 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.010463    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78c9b32e-faf1-442c-ba28-30d14902258e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n\" (UID: \"78c9b32e-faf1-442c-ba28-30d14902258e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.010634 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.010608    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78c9b32e-faf1-442c-ba28-30d14902258e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n\" (UID: \"78c9b32e-faf1-442c-ba28-30d14902258e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.010726 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.010669    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgnrr\" (UniqueName: \"kubernetes.io/projected/78c9b32e-faf1-442c-ba28-30d14902258e-kube-api-access-xgnrr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n\" (UID: \"78c9b32e-faf1-442c-ba28-30d14902258e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.010726 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.010707    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78c9b32e-faf1-442c-ba28-30d14902258e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n\" (UID: \"78c9b32e-faf1-442c-ba28-30d14902258e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.010821 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.010771    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78c9b32e-faf1-442c-ba28-30d14902258e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n\" (UID: \"78c9b32e-faf1-442c-ba28-30d14902258e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.010821 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.010797    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78c9b32e-faf1-442c-ba28-30d14902258e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n\" (UID: \"78c9b32e-faf1-442c-ba28-30d14902258e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.042901 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.042874    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:21.112134 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.112110    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78c9b32e-faf1-442c-ba28-30d14902258e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n\" (UID: \"78c9b32e-faf1-442c-ba28-30d14902258e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.112228 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.112146    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78c9b32e-faf1-442c-ba28-30d14902258e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n\" (UID: \"78c9b32e-faf1-442c-ba28-30d14902258e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.112299 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.112278    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnrr\" (UniqueName: \"kubernetes.io/projected/78c9b32e-faf1-442c-ba28-30d14902258e-kube-api-access-xgnrr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n\" (UID: \"78c9b32e-faf1-442c-ba28-30d14902258e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.112382 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.112322    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78c9b32e-faf1-442c-ba28-30d14902258e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n\" (UID: \"78c9b32e-faf1-442c-ba28-30d14902258e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.112446 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.112385    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78c9b32e-faf1-442c-ba28-30d14902258e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n\" (UID: \"78c9b32e-faf1-442c-ba28-30d14902258e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.112446 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.112410    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78c9b32e-faf1-442c-ba28-30d14902258e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n\" (UID: \"78c9b32e-faf1-442c-ba28-30d14902258e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.112541 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.112485    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78c9b32e-faf1-442c-ba28-30d14902258e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n\" (UID: \"78c9b32e-faf1-442c-ba28-30d14902258e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.112714 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.112691    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78c9b32e-faf1-442c-ba28-30d14902258e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n\" (UID: \"78c9b32e-faf1-442c-ba28-30d14902258e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.112785 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.112771    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78c9b32e-faf1-442c-ba28-30d14902258e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n\" (UID: \"78c9b32e-faf1-442c-ba28-30d14902258e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.114525 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.114505    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78c9b32e-faf1-442c-ba28-30d14902258e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n\" (UID: \"78c9b32e-faf1-442c-ba28-30d14902258e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.114588 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.114577    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78c9b32e-faf1-442c-ba28-30d14902258e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n\" (UID: \"78c9b32e-faf1-442c-ba28-30d14902258e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.131506 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.131487    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgnrr\" (UniqueName: \"kubernetes.io/projected/78c9b32e-faf1-442c-ba28-30d14902258e-kube-api-access-xgnrr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n\" (UID: \"78c9b32e-faf1-442c-ba28-30d14902258e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.255375 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.255307    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:21.545154 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.545128    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"]
Oct 30 19:29:21.546666 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:29:21.546644    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c9b32e_faf1_442c_ba28_30d14902258e.slice/crio-c2d8ae6d1b37d280a9d270402cd716f9d7f307312d1b48c85f92b3bcd1be7011 WatchSource:0}: Error finding container c2d8ae6d1b37d280a9d270402cd716f9d7f307312d1b48c85f92b3bcd1be7011: Status 404 returned error can't find the container with id c2d8ae6d1b37d280a9d270402cd716f9d7f307312d1b48c85f92b3bcd1be7011
Oct 30 19:29:21.691137 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.691104    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n" event={"ID":"78c9b32e-faf1-442c-ba28-30d14902258e","Type":"ContainerStarted","Data":"c2d8ae6d1b37d280a9d270402cd716f9d7f307312d1b48c85f92b3bcd1be7011"}
Oct 30 19:29:21.951627 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.951601    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2n6zs"]
Oct 30 19:29:21.954887 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.954856    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:21.972519 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.972498    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 30 19:29:21.979597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:21.979571    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2n6zs"]
Oct 30 19:29:22.020336 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.020309    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26jcc\" (UniqueName: \"kubernetes.io/projected/de5e94b0-0bc5-4fe0-afce-708bbbfd073b-kube-api-access-26jcc\") pod \"echo-simulator-kserve-64b6f8f96c-2n6zs\" (UID: \"de5e94b0-0bc5-4fe0-afce-708bbbfd073b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.020460 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.020347    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de5e94b0-0bc5-4fe0-afce-708bbbfd073b-home\") pod \"echo-simulator-kserve-64b6f8f96c-2n6zs\" (UID: \"de5e94b0-0bc5-4fe0-afce-708bbbfd073b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.020460 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.020456    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de5e94b0-0bc5-4fe0-afce-708bbbfd073b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2n6zs\" (UID: \"de5e94b0-0bc5-4fe0-afce-708bbbfd073b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.020544 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.020516    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de5e94b0-0bc5-4fe0-afce-708bbbfd073b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2n6zs\" (UID: \"de5e94b0-0bc5-4fe0-afce-708bbbfd073b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.020544 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.020540    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de5e94b0-0bc5-4fe0-afce-708bbbfd073b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2n6zs\" (UID: \"de5e94b0-0bc5-4fe0-afce-708bbbfd073b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.020613 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.020561    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de5e94b0-0bc5-4fe0-afce-708bbbfd073b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2n6zs\" (UID: \"de5e94b0-0bc5-4fe0-afce-708bbbfd073b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.064052 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.063992    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:22.121447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.121403    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de5e94b0-0bc5-4fe0-afce-708bbbfd073b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2n6zs\" (UID: \"de5e94b0-0bc5-4fe0-afce-708bbbfd073b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.121545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.121468    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de5e94b0-0bc5-4fe0-afce-708bbbfd073b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2n6zs\" (UID: \"de5e94b0-0bc5-4fe0-afce-708bbbfd073b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.121615 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.121529    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de5e94b0-0bc5-4fe0-afce-708bbbfd073b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2n6zs\" (UID: \"de5e94b0-0bc5-4fe0-afce-708bbbfd073b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.121681 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.121586    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26jcc\" (UniqueName: \"kubernetes.io/projected/de5e94b0-0bc5-4fe0-afce-708bbbfd073b-kube-api-access-26jcc\") pod \"echo-simulator-kserve-64b6f8f96c-2n6zs\" (UID: \"de5e94b0-0bc5-4fe0-afce-708bbbfd073b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.121746 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.121692    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de5e94b0-0bc5-4fe0-afce-708bbbfd073b-home\") pod \"echo-simulator-kserve-64b6f8f96c-2n6zs\" (UID: \"de5e94b0-0bc5-4fe0-afce-708bbbfd073b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.121857 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.121811    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de5e94b0-0bc5-4fe0-afce-708bbbfd073b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2n6zs\" (UID: \"de5e94b0-0bc5-4fe0-afce-708bbbfd073b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.121983 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.121956    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de5e94b0-0bc5-4fe0-afce-708bbbfd073b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2n6zs\" (UID: \"de5e94b0-0bc5-4fe0-afce-708bbbfd073b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.122064 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.122023    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de5e94b0-0bc5-4fe0-afce-708bbbfd073b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2n6zs\" (UID: \"de5e94b0-0bc5-4fe0-afce-708bbbfd073b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.122118 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.122070    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de5e94b0-0bc5-4fe0-afce-708bbbfd073b-home\") pod \"echo-simulator-kserve-64b6f8f96c-2n6zs\" (UID: \"de5e94b0-0bc5-4fe0-afce-708bbbfd073b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.123806 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.123780    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de5e94b0-0bc5-4fe0-afce-708bbbfd073b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2n6zs\" (UID: \"de5e94b0-0bc5-4fe0-afce-708bbbfd073b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.124201 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.124183    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de5e94b0-0bc5-4fe0-afce-708bbbfd073b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2n6zs\" (UID: \"de5e94b0-0bc5-4fe0-afce-708bbbfd073b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.140599 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.140578    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26jcc\" (UniqueName: \"kubernetes.io/projected/de5e94b0-0bc5-4fe0-afce-708bbbfd073b-kube-api-access-26jcc\") pod \"echo-simulator-kserve-64b6f8f96c-2n6zs\" (UID: \"de5e94b0-0bc5-4fe0-afce-708bbbfd073b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.268607 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.268583    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:22.557140 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.557070    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2n6zs"]
Oct 30 19:29:22.559635 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:29:22.559606    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5e94b0_0bc5_4fe0_afce_708bbbfd073b.slice/crio-62776f08b0a164aaf46ae250d33ceaf7b724dde69b1e1dd5e06dd6bc1a0db394 WatchSource:0}: Error finding container 62776f08b0a164aaf46ae250d33ceaf7b724dde69b1e1dd5e06dd6bc1a0db394: Status 404 returned error can't find the container with id 62776f08b0a164aaf46ae250d33ceaf7b724dde69b1e1dd5e06dd6bc1a0db394
Oct 30 19:29:22.697421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.697385    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n" event={"ID":"78c9b32e-faf1-442c-ba28-30d14902258e","Type":"ContainerStarted","Data":"20aeea6b6a3dc502784e26616c0a118aed68f218ad6c864a01f4adc1a71d5a6d"}
Oct 30 19:29:22.698618 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.698590    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs" event={"ID":"de5e94b0-0bc5-4fe0-afce-708bbbfd073b","Type":"ContainerStarted","Data":"62776f08b0a164aaf46ae250d33ceaf7b724dde69b1e1dd5e06dd6bc1a0db394"}
Oct 30 19:29:22.736803 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:22.736756    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n" podStartSLOduration=2.532686793 podStartE2EDuration="2.73674617s" totalImagesPullingTime="204.059377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:22.735957378 +0000 UTC m=+516.035219506" watchObservedRunningTime="2026-10-30 19:29:22.73674617 +0000 UTC m=+516.036008303"
Oct 30 19:29:23.256224 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.256193    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:23.292747 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.292308    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"]
Oct 30 19:29:23.299144 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.298397    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.326687 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.326642    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 19:29:23.329211 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.329183    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"]
Oct 30 19:29:23.401494 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.401461    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:23.433572 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.433534    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/038379f4-4d41-4502-8142-df1e91332286-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt\" (UID: \"038379f4-4d41-4502-8142-df1e91332286\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.433724 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.433594    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038379f4-4d41-4502-8142-df1e91332286-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt\" (UID: \"038379f4-4d41-4502-8142-df1e91332286\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.433724 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.433667    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkzvq\" (UniqueName: \"kubernetes.io/projected/038379f4-4d41-4502-8142-df1e91332286-kube-api-access-zkzvq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt\" (UID: \"038379f4-4d41-4502-8142-df1e91332286\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.433837 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.433773    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/038379f4-4d41-4502-8142-df1e91332286-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt\" (UID: \"038379f4-4d41-4502-8142-df1e91332286\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.433837 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.433810    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/038379f4-4d41-4502-8142-df1e91332286-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt\" (UID: \"038379f4-4d41-4502-8142-df1e91332286\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.433938 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.433851    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038379f4-4d41-4502-8142-df1e91332286-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt\" (UID: \"038379f4-4d41-4502-8142-df1e91332286\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.534887 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.534819    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/038379f4-4d41-4502-8142-df1e91332286-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt\" (UID: \"038379f4-4d41-4502-8142-df1e91332286\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.534887 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.534858    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038379f4-4d41-4502-8142-df1e91332286-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt\" (UID: \"038379f4-4d41-4502-8142-df1e91332286\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.535106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.534925    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/038379f4-4d41-4502-8142-df1e91332286-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt\" (UID: \"038379f4-4d41-4502-8142-df1e91332286\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.535106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.534961    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038379f4-4d41-4502-8142-df1e91332286-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt\" (UID: \"038379f4-4d41-4502-8142-df1e91332286\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.535106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.534978    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkzvq\" (UniqueName: \"kubernetes.io/projected/038379f4-4d41-4502-8142-df1e91332286-kube-api-access-zkzvq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt\" (UID: \"038379f4-4d41-4502-8142-df1e91332286\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.535106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.535029    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/038379f4-4d41-4502-8142-df1e91332286-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt\" (UID: \"038379f4-4d41-4502-8142-df1e91332286\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.535309 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.535156    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/038379f4-4d41-4502-8142-df1e91332286-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt\" (UID: \"038379f4-4d41-4502-8142-df1e91332286\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.535309 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.535252    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038379f4-4d41-4502-8142-df1e91332286-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt\" (UID: \"038379f4-4d41-4502-8142-df1e91332286\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.535401 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.535323    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/038379f4-4d41-4502-8142-df1e91332286-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt\" (UID: \"038379f4-4d41-4502-8142-df1e91332286\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.537017 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.536993    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/038379f4-4d41-4502-8142-df1e91332286-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt\" (UID: \"038379f4-4d41-4502-8142-df1e91332286\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.537267 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.537250    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038379f4-4d41-4502-8142-df1e91332286-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt\" (UID: \"038379f4-4d41-4502-8142-df1e91332286\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.554715 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.554692    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkzvq\" (UniqueName: \"kubernetes.io/projected/038379f4-4d41-4502-8142-df1e91332286-kube-api-access-zkzvq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt\" (UID: \"038379f4-4d41-4502-8142-df1e91332286\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.617388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.617291    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:23.908390 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:23.908345    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"]
Oct 30 19:29:23.912867 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:29:23.912840    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038379f4_4d41_4502_8142_df1e91332286.slice/crio-9ebb68d832eda736769e9fde55693ed22a2781b4598c7940fb96ab86ff49d127 WatchSource:0}: Error finding container 9ebb68d832eda736769e9fde55693ed22a2781b4598c7940fb96ab86ff49d127: Status 404 returned error can't find the container with id 9ebb68d832eda736769e9fde55693ed22a2781b4598c7940fb96ab86ff49d127
Oct 30 19:29:24.708851 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:24.708809    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt" event={"ID":"038379f4-4d41-4502-8142-df1e91332286","Type":"ContainerStarted","Data":"0045386fe46bcbf2d2b25a68f3254b52cb7b554ca8ae68c8322528546442a26f"}
Oct 30 19:29:24.709020 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:24.708872    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt" event={"ID":"038379f4-4d41-4502-8142-df1e91332286","Type":"ContainerStarted","Data":"9ebb68d832eda736769e9fde55693ed22a2781b4598c7940fb96ab86ff49d127"}
Oct 30 19:29:24.710378 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:24.710339    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs" event={"ID":"de5e94b0-0bc5-4fe0-afce-708bbbfd073b","Type":"ContainerStarted","Data":"e4791f7a9b49d7ed68956b75d379894fe586fd5126bb73980416814ea5c94f02"}
Oct 30 19:29:24.749328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:24.749066    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt" podStartSLOduration=1.5088469839999998 podStartE2EDuration="1.749050567s" totalImagesPullingTime="240.203583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:24.748648017 +0000 UTC m=+518.047910149" watchObservedRunningTime="2026-10-30 19:29:24.749050567 +0000 UTC m=+518.048312700"
Oct 30 19:29:24.788961 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:24.788919    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs" podStartSLOduration=2.346194785 podStartE2EDuration="3.788907981s" totalImagesPullingTime="1.442713196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:24.788544029 +0000 UTC m=+518.087806159" watchObservedRunningTime="2026-10-30 19:29:24.788907981 +0000 UTC m=+518.088170110"
Oct 30 19:29:25.269463 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:25.269435    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:25.618038 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:25.618015    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:26.770236 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.770201    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"]
Oct 30 19:29:26.773447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.773431    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:26.788157 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.788132    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 19:29:26.801053 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.801033    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"]
Oct 30 19:29:26.860915 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.860891    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd766cbc-5850-4181-95fa-a494ed7c8736-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc\" (UID: \"fd766cbc-5850-4181-95fa-a494ed7c8736\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:26.861020 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.860928    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd766cbc-5850-4181-95fa-a494ed7c8736-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc\" (UID: \"fd766cbc-5850-4181-95fa-a494ed7c8736\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:26.861098 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.861055    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdhpz\" (UniqueName: \"kubernetes.io/projected/fd766cbc-5850-4181-95fa-a494ed7c8736-kube-api-access-vdhpz\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc\" (UID: \"fd766cbc-5850-4181-95fa-a494ed7c8736\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:26.861098 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.861084    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd766cbc-5850-4181-95fa-a494ed7c8736-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc\" (UID: \"fd766cbc-5850-4181-95fa-a494ed7c8736\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:26.861221 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.861200    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd766cbc-5850-4181-95fa-a494ed7c8736-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc\" (UID: \"fd766cbc-5850-4181-95fa-a494ed7c8736\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:26.861354 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.861335    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd766cbc-5850-4181-95fa-a494ed7c8736-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc\" (UID: \"fd766cbc-5850-4181-95fa-a494ed7c8736\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:26.894954 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.894929    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:26.962450 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.962407    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhpz\" (UniqueName: \"kubernetes.io/projected/fd766cbc-5850-4181-95fa-a494ed7c8736-kube-api-access-vdhpz\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc\" (UID: \"fd766cbc-5850-4181-95fa-a494ed7c8736\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:26.962517 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.962457    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd766cbc-5850-4181-95fa-a494ed7c8736-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc\" (UID: \"fd766cbc-5850-4181-95fa-a494ed7c8736\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:26.962517 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.962490    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd766cbc-5850-4181-95fa-a494ed7c8736-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc\" (UID: \"fd766cbc-5850-4181-95fa-a494ed7c8736\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:26.962595 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.962532    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd766cbc-5850-4181-95fa-a494ed7c8736-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc\" (UID: \"fd766cbc-5850-4181-95fa-a494ed7c8736\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:26.962595 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.962565    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd766cbc-5850-4181-95fa-a494ed7c8736-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc\" (UID: \"fd766cbc-5850-4181-95fa-a494ed7c8736\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:26.962595 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.962581    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd766cbc-5850-4181-95fa-a494ed7c8736-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc\" (UID: \"fd766cbc-5850-4181-95fa-a494ed7c8736\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:26.962875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.962855    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd766cbc-5850-4181-95fa-a494ed7c8736-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc\" (UID: \"fd766cbc-5850-4181-95fa-a494ed7c8736\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:26.962950 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.962929    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd766cbc-5850-4181-95fa-a494ed7c8736-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc\" (UID: \"fd766cbc-5850-4181-95fa-a494ed7c8736\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:26.963000 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.962984    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd766cbc-5850-4181-95fa-a494ed7c8736-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc\" (UID: \"fd766cbc-5850-4181-95fa-a494ed7c8736\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:26.964809 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.964782    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd766cbc-5850-4181-95fa-a494ed7c8736-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc\" (UID: \"fd766cbc-5850-4181-95fa-a494ed7c8736\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:26.964963 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.964943    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd766cbc-5850-4181-95fa-a494ed7c8736-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc\" (UID: \"fd766cbc-5850-4181-95fa-a494ed7c8736\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:26.981416 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:26.981390    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhpz\" (UniqueName: \"kubernetes.io/projected/fd766cbc-5850-4181-95fa-a494ed7c8736-kube-api-access-vdhpz\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc\" (UID: \"fd766cbc-5850-4181-95fa-a494ed7c8736\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:27.084140 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:27.084086    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:27.380701 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:27.380677    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"]
Oct 30 19:29:27.382208 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:29:27.382180    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd766cbc_5850_4181_95fa_a494ed7c8736.slice/crio-9e5446846d1def4c8ec05741fd2d5940fd96241f11c14bc0422bf57cc37b452e WatchSource:0}: Error finding container 9e5446846d1def4c8ec05741fd2d5940fd96241f11c14bc0422bf57cc37b452e: Status 404 returned error can't find the container with id 9e5446846d1def4c8ec05741fd2d5940fd96241f11c14bc0422bf57cc37b452e
Oct 30 19:29:27.726304 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:27.726269    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" event={"ID":"fd766cbc-5850-4181-95fa-a494ed7c8736","Type":"ContainerStarted","Data":"984fb1918f0a0be0aca79f01e4bb83b7940648dd0e8ae24d998750da3b6f895c"}
Oct 30 19:29:27.726451 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:27.726323    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" event={"ID":"fd766cbc-5850-4181-95fa-a494ed7c8736","Type":"ContainerStarted","Data":"9e5446846d1def4c8ec05741fd2d5940fd96241f11c14bc0422bf57cc37b452e"}
Oct 30 19:29:27.765908 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:27.765862    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podStartSLOduration=1.535202368 podStartE2EDuration="1.765849084s" totalImagesPullingTime="230.646716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:27.764430521 +0000 UTC m=+521.063692652" watchObservedRunningTime="2026-10-30 19:29:27.765849084 +0000 UTC m=+521.065111207"
Oct 30 19:29:28.084731 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:28.084664    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:28.931272 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:28.931239    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:28.943613 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:28.943589    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:29.744273 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:29.744245    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xzskw"
Oct 30 19:29:30.302954 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:30.302915    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:30.473101 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:30.473065    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:31.255797 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.255758    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:31.267733 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.267706    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:31.596173 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.596095    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"]
Oct 30 19:29:31.599508 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.599485    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.619335 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.619304    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 19:29:31.630681 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.630658    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"]
Oct 30 19:29:31.706642 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.706617    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45dec9f8-1002-4ccc-b536-d43e3c3cfbb4-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k2ndr\" (UID: \"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.706755 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.706651    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6d2j\" (UniqueName: \"kubernetes.io/projected/45dec9f8-1002-4ccc-b536-d43e3c3cfbb4-kube-api-access-b6d2j\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k2ndr\" (UID: \"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.706755 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.706688    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45dec9f8-1002-4ccc-b536-d43e3c3cfbb4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k2ndr\" (UID: \"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.706835 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.706795    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45dec9f8-1002-4ccc-b536-d43e3c3cfbb4-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k2ndr\" (UID: \"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.706880 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.706838    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45dec9f8-1002-4ccc-b536-d43e3c3cfbb4-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k2ndr\" (UID: \"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.706880 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.706861    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45dec9f8-1002-4ccc-b536-d43e3c3cfbb4-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k2ndr\" (UID: \"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.738226 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.738202    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:31.753808 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.753786    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n"
Oct 30 19:29:31.807623 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.807600    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45dec9f8-1002-4ccc-b536-d43e3c3cfbb4-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k2ndr\" (UID: \"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.807732 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.807631    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45dec9f8-1002-4ccc-b536-d43e3c3cfbb4-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k2ndr\" (UID: \"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.807732 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.807654    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45dec9f8-1002-4ccc-b536-d43e3c3cfbb4-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k2ndr\" (UID: \"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.807732 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.807672    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6d2j\" (UniqueName: \"kubernetes.io/projected/45dec9f8-1002-4ccc-b536-d43e3c3cfbb4-kube-api-access-b6d2j\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k2ndr\" (UID: \"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.807732 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.807718    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45dec9f8-1002-4ccc-b536-d43e3c3cfbb4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k2ndr\" (UID: \"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.807965 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.807795    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45dec9f8-1002-4ccc-b536-d43e3c3cfbb4-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k2ndr\" (UID: \"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.808162 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.808139    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45dec9f8-1002-4ccc-b536-d43e3c3cfbb4-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k2ndr\" (UID: \"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.808315 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.808200    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45dec9f8-1002-4ccc-b536-d43e3c3cfbb4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k2ndr\" (UID: \"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.808435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.808296    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45dec9f8-1002-4ccc-b536-d43e3c3cfbb4-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k2ndr\" (UID: \"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.810057 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.810033    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45dec9f8-1002-4ccc-b536-d43e3c3cfbb4-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k2ndr\" (UID: \"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.810322 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.810303    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45dec9f8-1002-4ccc-b536-d43e3c3cfbb4-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k2ndr\" (UID: \"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.827135 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.827111    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6d2j\" (UniqueName: \"kubernetes.io/projected/45dec9f8-1002-4ccc-b536-d43e3c3cfbb4-kube-api-access-b6d2j\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k2ndr\" (UID: \"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:31.914439 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:31.914413    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:32.205290 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:32.205256    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"]
Oct 30 19:29:32.205962 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:29:32.205935    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45dec9f8_1002_4ccc_b536_d43e3c3cfbb4.slice/crio-fda3e224e044cd5e1613d7f7ae0d03ebf4c7dd03cc2ab5be23435a1a3b8a14d6 WatchSource:0}: Error finding container fda3e224e044cd5e1613d7f7ae0d03ebf4c7dd03cc2ab5be23435a1a3b8a14d6: Status 404 returned error can't find the container with id fda3e224e044cd5e1613d7f7ae0d03ebf4c7dd03cc2ab5be23435a1a3b8a14d6
Oct 30 19:29:32.269395 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:32.269347    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:32.281597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:32.281578    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:32.747731 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:32.747636    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr" event={"ID":"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4","Type":"ContainerStarted","Data":"3940839c971731a71b41177347dfdaf3bdf0c96e43029f99f3aff8ee5b26135d"}
Oct 30 19:29:32.747731 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:32.747703    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr" event={"ID":"45dec9f8-1002-4ccc-b536-d43e3c3cfbb4","Type":"ContainerStarted","Data":"fda3e224e044cd5e1613d7f7ae0d03ebf4c7dd03cc2ab5be23435a1a3b8a14d6"}
Oct 30 19:29:32.758872 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:32.758843    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-2n6zs"
Oct 30 19:29:32.787826 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:32.787781    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr" podStartSLOduration=1.5789459959999999 podStartE2EDuration="1.787770461s" totalImagesPullingTime="208.824465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:32.787416794 +0000 UTC m=+526.086678922" watchObservedRunningTime="2026-10-30 19:29:32.787770461 +0000 UTC m=+526.087032589"
Oct 30 19:29:32.915332 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:32.915290    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:33.531597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:33.531561    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:33.617674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:33.617638    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:33.629966 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:33.629947    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:33.763406 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:33.763374    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt"
Oct 30 19:29:36.371947 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:29:36.371913    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:29:37.084340 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:37.084307    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:37.096746 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:37.096723    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:37.780014 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:37.779987    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:29:38.243783 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.243755    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"]
Oct 30 19:29:38.247050 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.247027    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.261537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.261516    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:29:38.273332 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.273310    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-q4g2l\""
Oct 30 19:29:38.275177 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.275156    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"]
Oct 30 19:29:38.356607 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.356583    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m7rz\" (UniqueName: \"kubernetes.io/projected/75fe15f5-086d-455e-af75-a99955780e72-kube-api-access-2m7rz\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.356737 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.356617    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75fe15f5-086d-455e-af75-a99955780e72-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.356737 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.356635    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75fe15f5-086d-455e-af75-a99955780e72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.356737 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.356661    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75fe15f5-086d-455e-af75-a99955780e72-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.356846 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.356801    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75fe15f5-086d-455e-af75-a99955780e72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.356846 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.356830    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75fe15f5-086d-455e-af75-a99955780e72-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.356902 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.356848    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75fe15f5-086d-455e-af75-a99955780e72-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.356944 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.356931    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75fe15f5-086d-455e-af75-a99955780e72-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.356976 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.356958    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75fe15f5-086d-455e-af75-a99955780e72-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.457458 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.457431    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75fe15f5-086d-455e-af75-a99955780e72-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.457582 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.457467    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75fe15f5-086d-455e-af75-a99955780e72-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.457582 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.457486    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m7rz\" (UniqueName: \"kubernetes.io/projected/75fe15f5-086d-455e-af75-a99955780e72-kube-api-access-2m7rz\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.457582 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.457509    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75fe15f5-086d-455e-af75-a99955780e72-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.457582 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.457534    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75fe15f5-086d-455e-af75-a99955780e72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.457582 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.457572    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75fe15f5-086d-455e-af75-a99955780e72-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.457865 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.457608    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75fe15f5-086d-455e-af75-a99955780e72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.457929 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.457863    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75fe15f5-086d-455e-af75-a99955780e72-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.457929 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.457884    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75fe15f5-086d-455e-af75-a99955780e72-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.457929 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.457911    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75fe15f5-086d-455e-af75-a99955780e72-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.458044 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.457939    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75fe15f5-086d-455e-af75-a99955780e72-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.458227 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.458206    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75fe15f5-086d-455e-af75-a99955780e72-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.458331 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.458275    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75fe15f5-086d-455e-af75-a99955780e72-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.458412 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.458353    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75fe15f5-086d-455e-af75-a99955780e72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.460342 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.460314    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75fe15f5-086d-455e-af75-a99955780e72-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.460560 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.460545    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75fe15f5-086d-455e-af75-a99955780e72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.476968 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.476933    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75fe15f5-086d-455e-af75-a99955780e72-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.477177 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.477158    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m7rz\" (UniqueName: \"kubernetes.io/projected/75fe15f5-086d-455e-af75-a99955780e72-kube-api-access-2m7rz\") pod \"maas-default-gateway-openshift-default-5bc884dd87-rflrd\" (UID: \"75fe15f5-086d-455e-af75-a99955780e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.559834 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.559770    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:38.874534 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:38.874512    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"]
Oct 30 19:29:38.875390 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:29:38.875351    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75fe15f5_086d_455e_af75_a99955780e72.slice/crio-7fe98636916fbc3be8c22bc265aff94ecbfd4b446b40c03e2a052669202d1f5e WatchSource:0}: Error finding container 7fe98636916fbc3be8c22bc265aff94ecbfd4b446b40c03e2a052669202d1f5e: Status 404 returned error can't find the container with id 7fe98636916fbc3be8c22bc265aff94ecbfd4b446b40c03e2a052669202d1f5e
Oct 30 19:29:39.779281 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:39.779246    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd" event={"ID":"75fe15f5-086d-455e-af75-a99955780e72","Type":"ContainerStarted","Data":"7fe98636916fbc3be8c22bc265aff94ecbfd4b446b40c03e2a052669202d1f5e"}
Oct 30 19:29:41.461537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:41.461506    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:29:41.461803 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:41.461576    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:29:41.461803 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:41.461610    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:29:41.789545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:41.789469    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd" event={"ID":"75fe15f5-086d-455e-af75-a99955780e72","Type":"ContainerStarted","Data":"cca927a48ea58b7b3b2577569788ae4abbc1fb5bb9b94598409ea040f48145f9"}
Oct 30 19:29:41.858836 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:41.858787    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd" podStartSLOduration=1.275429827 podStartE2EDuration="3.858775643s" totalImagesPullingTime="2.583345816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:41.846504258 +0000 UTC m=+535.145766386" watchObservedRunningTime="2026-10-30 19:29:41.858775643 +0000 UTC m=+535.158037771"
Oct 30 19:29:41.915582 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:41.915547    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:41.936388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:41.934763    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:42.560479 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:42.560438    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:42.561934 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:42.561911    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rflrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 19:29:42.562020 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:42.561961    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd" podUID="75fe15f5-086d-455e-af75-a99955780e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 19:29:42.816440 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:42.804428    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k2ndr"
Oct 30 19:29:43.560999 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:43.560967    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rflrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 19:29:43.561425 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:43.561019    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd" podUID="75fe15f5-086d-455e-af75-a99955780e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 19:29:44.560932 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:44.560895    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rflrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 19:29:44.561120 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:44.560952    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd" podUID="75fe15f5-086d-455e-af75-a99955780e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 19:29:45.560724 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:45.560694    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rflrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 19:29:45.560871 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:45.560741    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd" podUID="75fe15f5-086d-455e-af75-a99955780e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 19:29:46.561066 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:46.561029    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rflrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 19:29:46.561477 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:46.561082    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd" podUID="75fe15f5-086d-455e-af75-a99955780e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 19:29:47.520652 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.520618    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz"]
Oct 30 19:29:47.528955 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.528930    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz"
Oct 30 19:29:47.546163 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.546131    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz"]
Oct 30 19:29:47.556342 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.556319    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-8s4q9\""
Oct 30 19:29:47.556536 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.556386    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 19:29:47.556536 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.556509    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 19:29:47.556536 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.556444    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 19:29:47.556536 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.556447    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 19:29:47.559762 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.559736    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:47.560042 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.560021    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rflrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 19:29:47.560112 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.560065    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd" podUID="75fe15f5-086d-455e-af75-a99955780e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 19:29:47.661206 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.661179    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htcxj\" (UniqueName: \"kubernetes.io/projected/935ffff7-de98-4e0b-98ed-79e4016dce59-kube-api-access-htcxj\") pod \"ai-gateway-controller-6cf7bcbfb6-p8pgz\" (UID: \"935ffff7-de98-4e0b-98ed-79e4016dce59\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz"
Oct 30 19:29:47.661574 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.661249    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/935ffff7-de98-4e0b-98ed-79e4016dce59-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-p8pgz\" (UID: \"935ffff7-de98-4e0b-98ed-79e4016dce59\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz"
Oct 30 19:29:47.719337 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.719306    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:47.762400 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.762354    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htcxj\" (UniqueName: \"kubernetes.io/projected/935ffff7-de98-4e0b-98ed-79e4016dce59-kube-api-access-htcxj\") pod \"ai-gateway-controller-6cf7bcbfb6-p8pgz\" (UID: \"935ffff7-de98-4e0b-98ed-79e4016dce59\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz"
Oct 30 19:29:47.762547 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.762450    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/935ffff7-de98-4e0b-98ed-79e4016dce59-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-p8pgz\" (UID: \"935ffff7-de98-4e0b-98ed-79e4016dce59\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz"
Oct 30 19:29:47.764968 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.764942    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/935ffff7-de98-4e0b-98ed-79e4016dce59-metrics-certs\") pod \"ai-gateway-controller-6cf7bcbfb6-p8pgz\" (UID: \"935ffff7-de98-4e0b-98ed-79e4016dce59\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz"
Oct 30 19:29:47.782244 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.782189    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htcxj\" (UniqueName: \"kubernetes.io/projected/935ffff7-de98-4e0b-98ed-79e4016dce59-kube-api-access-htcxj\") pod \"ai-gateway-controller-6cf7bcbfb6-p8pgz\" (UID: \"935ffff7-de98-4e0b-98ed-79e4016dce59\") " pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz"
Oct 30 19:29:47.841699 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:47.841673    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz"
Oct 30 19:29:48.140790 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:29:48.140761    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935ffff7_de98_4e0b_98ed_79e4016dce59.slice/crio-35226de0f22ff46a8d3cda2b9e0ec3a3b80eca7c2eb1ea3b0bcd33d7b09aeb01 WatchSource:0}: Error finding container 35226de0f22ff46a8d3cda2b9e0ec3a3b80eca7c2eb1ea3b0bcd33d7b09aeb01: Status 404 returned error can't find the container with id 35226de0f22ff46a8d3cda2b9e0ec3a3b80eca7c2eb1ea3b0bcd33d7b09aeb01
Oct 30 19:29:48.148595 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:48.148571    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz"]
Oct 30 19:29:48.560687 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:48.560612    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-rflrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 19:29:48.560687 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:48.560671    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd" podUID="75fe15f5-086d-455e-af75-a99955780e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 19:29:48.844106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:48.844013    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz" event={"ID":"935ffff7-de98-4e0b-98ed-79e4016dce59","Type":"ContainerStarted","Data":"35226de0f22ff46a8d3cda2b9e0ec3a3b80eca7c2eb1ea3b0bcd33d7b09aeb01"}
Oct 30 19:29:49.593958 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:49.593917    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:49.594476 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:49.594456    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:49.595377 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:49.595344    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-rflrd"
Oct 30 19:29:49.849661 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:49.849583    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz" event={"ID":"935ffff7-de98-4e0b-98ed-79e4016dce59","Type":"ContainerStarted","Data":"ecf575a56c3275172ae8002ebf623c5983ce322a27bdf1752542fab7d6a6861a"}
Oct 30 19:29:49.850067 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:49.849676    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz"
Oct 30 19:29:49.889862 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:49.889806    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz" podStartSLOduration=1.414924753 podStartE2EDuration="2.88979486s" totalImagesPullingTime="1.474870107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:49.888355497 +0000 UTC m=+543.187617625" watchObservedRunningTime="2026-10-30 19:29:49.88979486 +0000 UTC m=+543.189056989"
Oct 30 19:29:51.946615 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:51.946583    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:52.087912 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:52.087878    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7p7sg"]
Oct 30 19:29:52.093034 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:52.093013    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7p7sg"
Oct 30 19:29:52.110248 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:52.110223    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7p7sg"]
Oct 30 19:29:52.121092 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:52.121071    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l54cm\""
Oct 30 19:29:52.141640 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:52.141607    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:29:52.186614 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:52.186579    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 19:29:52.192106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:52.192082    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17538073-ba9e-44e4-98b1-80a78223f37d-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7p7sg\" (UID: \"17538073-ba9e-44e4-98b1-80a78223f37d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7p7sg"
Oct 30 19:29:52.192212 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:52.192137    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx9s7\" (UniqueName: \"kubernetes.io/projected/17538073-ba9e-44e4-98b1-80a78223f37d-kube-api-access-vx9s7\") pod \"payload-pre-processing-7b98bb678c-7p7sg\" (UID: \"17538073-ba9e-44e4-98b1-80a78223f37d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7p7sg"
Oct 30 19:29:52.293175 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:52.293090    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx9s7\" (UniqueName: \"kubernetes.io/projected/17538073-ba9e-44e4-98b1-80a78223f37d-kube-api-access-vx9s7\") pod \"payload-pre-processing-7b98bb678c-7p7sg\" (UID: \"17538073-ba9e-44e4-98b1-80a78223f37d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7p7sg"
Oct 30 19:29:52.293175 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:52.293161    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17538073-ba9e-44e4-98b1-80a78223f37d-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7p7sg\" (UID: \"17538073-ba9e-44e4-98b1-80a78223f37d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7p7sg"
Oct 30 19:29:52.293904 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:52.293882    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17538073-ba9e-44e4-98b1-80a78223f37d-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7p7sg\" (UID: \"17538073-ba9e-44e4-98b1-80a78223f37d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7p7sg"
Oct 30 19:29:52.312630 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:52.312602    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx9s7\" (UniqueName: \"kubernetes.io/projected/17538073-ba9e-44e4-98b1-80a78223f37d-kube-api-access-vx9s7\") pod \"payload-pre-processing-7b98bb678c-7p7sg\" (UID: \"17538073-ba9e-44e4-98b1-80a78223f37d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7p7sg"
Oct 30 19:29:52.405505 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:52.405477    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7p7sg"
Oct 30 19:29:52.704171 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:52.704140    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7p7sg"]
Oct 30 19:29:52.705435 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:29:52.705405    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17538073_ba9e_44e4_98b1_80a78223f37d.slice/crio-9cfdfa3e93d88209fc083fa8878e93130c4b31167cb9124cb3bc328916e583fa WatchSource:0}: Error finding container 9cfdfa3e93d88209fc083fa8878e93130c4b31167cb9124cb3bc328916e583fa: Status 404 returned error can't find the container with id 9cfdfa3e93d88209fc083fa8878e93130c4b31167cb9124cb3bc328916e583fa
Oct 30 19:29:52.862569 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:52.862532    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7p7sg" event={"ID":"17538073-ba9e-44e4-98b1-80a78223f37d","Type":"ContainerStarted","Data":"9cfdfa3e93d88209fc083fa8878e93130c4b31167cb9124cb3bc328916e583fa"}
Oct 30 19:29:53.305454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.305354    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"]
Oct 30 19:29:53.310178 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.309651    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.340437 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.340065    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"]
Oct 30 19:29:53.401958 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.401923    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbshd\" (UniqueName: \"kubernetes.io/projected/8cbf06ec-e967-4cc4-9d06-6629e3be62da-kube-api-access-sbshd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.402138 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.401985    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cbf06ec-e967-4cc4-9d06-6629e3be62da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.402138 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.402112    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cbf06ec-e967-4cc4-9d06-6629e3be62da-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.402273 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.402142    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cbf06ec-e967-4cc4-9d06-6629e3be62da-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.402273 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.402176    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cbf06ec-e967-4cc4-9d06-6629e3be62da-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.402273 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.402210    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbf06ec-e967-4cc4-9d06-6629e3be62da-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.402475 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.402280    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cbf06ec-e967-4cc4-9d06-6629e3be62da-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.402475 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.402308    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbf06ec-e967-4cc4-9d06-6629e3be62da-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.402475 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.402357    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cbf06ec-e967-4cc4-9d06-6629e3be62da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.503686 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.503652    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbshd\" (UniqueName: \"kubernetes.io/projected/8cbf06ec-e967-4cc4-9d06-6629e3be62da-kube-api-access-sbshd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.503850 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.503695    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cbf06ec-e967-4cc4-9d06-6629e3be62da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.503850 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.503719    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cbf06ec-e967-4cc4-9d06-6629e3be62da-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.503850 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.503744    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cbf06ec-e967-4cc4-9d06-6629e3be62da-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.503850 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.503770    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cbf06ec-e967-4cc4-9d06-6629e3be62da-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.503850 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.503802    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbf06ec-e967-4cc4-9d06-6629e3be62da-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.504054 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.503859    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cbf06ec-e967-4cc4-9d06-6629e3be62da-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.504054 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.503886    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbf06ec-e967-4cc4-9d06-6629e3be62da-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.504054 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.503941    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cbf06ec-e967-4cc4-9d06-6629e3be62da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.504311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.504280    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cbf06ec-e967-4cc4-9d06-6629e3be62da-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.523391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.504488    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cbf06ec-e967-4cc4-9d06-6629e3be62da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.523391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.504560    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cbf06ec-e967-4cc4-9d06-6629e3be62da-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.523391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.504494    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbf06ec-e967-4cc4-9d06-6629e3be62da-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.523391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.504660    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cbf06ec-e967-4cc4-9d06-6629e3be62da-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.523391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.509255    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cbf06ec-e967-4cc4-9d06-6629e3be62da-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.523391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.509741    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cbf06ec-e967-4cc4-9d06-6629e3be62da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.524679 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.524650    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cbf06ec-e967-4cc4-9d06-6629e3be62da-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.526422 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.526393    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbshd\" (UniqueName: \"kubernetes.io/projected/8cbf06ec-e967-4cc4-9d06-6629e3be62da-kube-api-access-sbshd\") pod \"maas-default-gateway-openshift-default-5bc884dd87-28vt6\" (UID: \"8cbf06ec-e967-4cc4-9d06-6629e3be62da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.630223 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.629905    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:53.970245 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.970105    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"]
Oct 30 19:29:53.973649 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:29:53.973617    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbf06ec_e967_4cc4_9d06_6629e3be62da.slice/crio-8e19d1eb5f87abb5c4b4ec4f76955ccac91b8b89cee92693d1aea50260187e3a WatchSource:0}: Error finding container 8e19d1eb5f87abb5c4b4ec4f76955ccac91b8b89cee92693d1aea50260187e3a: Status 404 returned error can't find the container with id 8e19d1eb5f87abb5c4b4ec4f76955ccac91b8b89cee92693d1aea50260187e3a
Oct 30 19:29:53.976139 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.976099    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:29:53.976223 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.976187    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:29:53.976262 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:53.976220    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:29:54.874673 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:54.874635    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6" event={"ID":"8cbf06ec-e967-4cc4-9d06-6629e3be62da","Type":"ContainerStarted","Data":"e90dfa1f355398a1351811cc8558808e1b027ef0fdc96bd78fe9c0dbb4183829"}
Oct 30 19:29:54.875149 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:54.874689    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6" event={"ID":"8cbf06ec-e967-4cc4-9d06-6629e3be62da","Type":"ContainerStarted","Data":"8e19d1eb5f87abb5c4b4ec4f76955ccac91b8b89cee92693d1aea50260187e3a"}
Oct 30 19:29:54.945256 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:54.945201    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6" podStartSLOduration=1.945180872 podStartE2EDuration="1.945180872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:54.932394987 +0000 UTC m=+548.231657116" watchObservedRunningTime="2026-10-30 19:29:54.945180872 +0000 UTC m=+548.244443003"
Oct 30 19:29:55.631161 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:55.631123    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:29:55.632483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:55.632455    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-28vt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 19:29:55.632582 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:55.632546    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6" podUID="8cbf06ec-e967-4cc4-9d06-6629e3be62da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 19:29:55.880824 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:55.880787    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7p7sg" event={"ID":"17538073-ba9e-44e4-98b1-80a78223f37d","Type":"ContainerStarted","Data":"f9e5a32a5e775d95a08816e447262fcec5430e4e809bd476668067972d4870bc"}
Oct 30 19:29:55.881320 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:55.880882    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7p7sg"
Oct 30 19:29:55.927384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:55.924669    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7p7sg" podStartSLOduration=1.396270943 podStartE2EDuration="3.92465058s" totalImagesPullingTime="2.528379637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:55.922502806 +0000 UTC m=+549.221764937" watchObservedRunningTime="2026-10-30 19:29:55.92465058 +0000 UTC m=+549.223912712"
Oct 30 19:29:56.630931 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:56.630896    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-28vt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 19:29:56.631104 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:56.630948    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6" podUID="8cbf06ec-e967-4cc4-9d06-6629e3be62da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 19:29:57.632267 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:57.632233    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-28vt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 19:29:57.632718 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:57.632289    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6" podUID="8cbf06ec-e967-4cc4-9d06-6629e3be62da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 19:29:58.630550 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:58.630513    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-28vt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 19:29:58.630722 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:58.630574    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6" podUID="8cbf06ec-e967-4cc4-9d06-6629e3be62da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 19:29:59.631307 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:59.631266    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-28vt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 19:29:59.631808 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:29:59.631327    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6" podUID="8cbf06ec-e967-4cc4-9d06-6629e3be62da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 19:30:00.631546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:00.631501    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-28vt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 30 19:30:00.631952 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:00.631562    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6" podUID="8cbf06ec-e967-4cc4-9d06-6629e3be62da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 30 19:30:00.856406 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:00.856355    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz"
Oct 30 19:30:01.136304 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:01.136273    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-744cf976bc-l8stx"]
Oct 30 19:30:01.139439 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:01.139421    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-744cf976bc-l8stx"
Oct 30 19:30:01.158285 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:01.158264    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-744cf976bc-l8stx"]
Oct 30 19:30:01.183110 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:01.183073    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm62j\" (UniqueName: \"kubernetes.io/projected/916569a8-b32b-410a-b619-cd522570fbd9-kube-api-access-vm62j\") pod \"ai-gateway-controller-744cf976bc-l8stx\" (UID: \"916569a8-b32b-410a-b619-cd522570fbd9\") " pod="opendatahub/ai-gateway-controller-744cf976bc-l8stx"
Oct 30 19:30:01.183274 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:01.183256    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/916569a8-b32b-410a-b619-cd522570fbd9-metrics-certs\") pod \"ai-gateway-controller-744cf976bc-l8stx\" (UID: \"916569a8-b32b-410a-b619-cd522570fbd9\") " pod="opendatahub/ai-gateway-controller-744cf976bc-l8stx"
Oct 30 19:30:01.284025 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:01.283994    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm62j\" (UniqueName: \"kubernetes.io/projected/916569a8-b32b-410a-b619-cd522570fbd9-kube-api-access-vm62j\") pod \"ai-gateway-controller-744cf976bc-l8stx\" (UID: \"916569a8-b32b-410a-b619-cd522570fbd9\") " pod="opendatahub/ai-gateway-controller-744cf976bc-l8stx"
Oct 30 19:30:01.284155 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:01.284099    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/916569a8-b32b-410a-b619-cd522570fbd9-metrics-certs\") pod \"ai-gateway-controller-744cf976bc-l8stx\" (UID: \"916569a8-b32b-410a-b619-cd522570fbd9\") " pod="opendatahub/ai-gateway-controller-744cf976bc-l8stx"
Oct 30 19:30:01.286465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:01.286442    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/916569a8-b32b-410a-b619-cd522570fbd9-metrics-certs\") pod \"ai-gateway-controller-744cf976bc-l8stx\" (UID: \"916569a8-b32b-410a-b619-cd522570fbd9\") " pod="opendatahub/ai-gateway-controller-744cf976bc-l8stx"
Oct 30 19:30:01.303580 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:01.303556    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm62j\" (UniqueName: \"kubernetes.io/projected/916569a8-b32b-410a-b619-cd522570fbd9-kube-api-access-vm62j\") pod \"ai-gateway-controller-744cf976bc-l8stx\" (UID: \"916569a8-b32b-410a-b619-cd522570fbd9\") " pod="opendatahub/ai-gateway-controller-744cf976bc-l8stx"
Oct 30 19:30:01.452133 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:01.452071    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-744cf976bc-l8stx"
Oct 30 19:30:01.740557 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:01.740531    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-744cf976bc-l8stx"]
Oct 30 19:30:01.743136 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:30:01.743089    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916569a8_b32b_410a_b619_cd522570fbd9.slice/crio-66f44ea25c177d3da2822762cd7c0249aaee759a7ab78d8ab92297226e4e1dd6 WatchSource:0}: Error finding container 66f44ea25c177d3da2822762cd7c0249aaee759a7ab78d8ab92297226e4e1dd6: Status 404 returned error can't find the container with id 66f44ea25c177d3da2822762cd7c0249aaee759a7ab78d8ab92297226e4e1dd6
Oct 30 19:30:01.906255 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:01.906226    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-744cf976bc-l8stx" event={"ID":"916569a8-b32b-410a-b619-cd522570fbd9","Type":"ContainerStarted","Data":"66f44ea25c177d3da2822762cd7c0249aaee759a7ab78d8ab92297226e4e1dd6"}
Oct 30 19:30:02.351272 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:02.351243    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:30:02.351749 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:02.351724    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:30:02.352519 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:02.352498    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6"
Oct 30 19:30:02.911819 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:02.911783    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-744cf976bc-l8stx" event={"ID":"916569a8-b32b-410a-b619-cd522570fbd9","Type":"ContainerStarted","Data":"50ee5384deeba98284d4f91e33435c946f5ae75f4a6227d59e6f01900aff62c6"}
Oct 30 19:30:02.952207 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:02.952162    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-744cf976bc-l8stx" podStartSLOduration=1.565787168 podStartE2EDuration="1.952148472s" totalImagesPullingTime="386.361304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:02.950682657 +0000 UTC m=+556.249944786" watchObservedRunningTime="2026-10-30 19:30:02.952148472 +0000 UTC m=+556.251410600"
Oct 30 19:30:03.916425 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:03.916384    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-744cf976bc-l8stx"
Oct 30 19:30:06.372558 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:30:06.372460    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:30:06.887578 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:06.887556    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7p7sg"
Oct 30 19:30:08.343625 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.343592    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"]
Oct 30 19:30:08.346930 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.346914    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.372857 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.372830    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"]
Oct 30 19:30:08.439421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.439388    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1c940f3-a6ba-45e5-bb26-2997810c97b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.439581 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.439428    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1c940f3-a6ba-45e5-bb26-2997810c97b6-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.439581 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.439508    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc7td\" (UniqueName: \"kubernetes.io/projected/f1c940f3-a6ba-45e5-bb26-2997810c97b6-kube-api-access-zc7td\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.439581 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.439563    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1c940f3-a6ba-45e5-bb26-2997810c97b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.439704 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.439581    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c940f3-a6ba-45e5-bb26-2997810c97b6-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.439704 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.439598    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1c940f3-a6ba-45e5-bb26-2997810c97b6-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.439704 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.439636    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1c940f3-a6ba-45e5-bb26-2997810c97b6-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.439704 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.439696    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c940f3-a6ba-45e5-bb26-2997810c97b6-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.439833 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.439745    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1c940f3-a6ba-45e5-bb26-2997810c97b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.540356 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.540324    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1c940f3-a6ba-45e5-bb26-2997810c97b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.540356 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.540356    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1c940f3-a6ba-45e5-bb26-2997810c97b6-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.540579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.540399    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc7td\" (UniqueName: \"kubernetes.io/projected/f1c940f3-a6ba-45e5-bb26-2997810c97b6-kube-api-access-zc7td\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.540579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.540420    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1c940f3-a6ba-45e5-bb26-2997810c97b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.540579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.540437    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c940f3-a6ba-45e5-bb26-2997810c97b6-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.540579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.540452    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1c940f3-a6ba-45e5-bb26-2997810c97b6-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.540579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.540470    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1c940f3-a6ba-45e5-bb26-2997810c97b6-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.540579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.540501    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c940f3-a6ba-45e5-bb26-2997810c97b6-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.540579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.540562    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1c940f3-a6ba-45e5-bb26-2997810c97b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.540938 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.540854    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c940f3-a6ba-45e5-bb26-2997810c97b6-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.540938 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.540881    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1c940f3-a6ba-45e5-bb26-2997810c97b6-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.541045 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.541008    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c940f3-a6ba-45e5-bb26-2997810c97b6-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.541099 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.541041    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1c940f3-a6ba-45e5-bb26-2997810c97b6-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.541288 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.541265    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1c940f3-a6ba-45e5-bb26-2997810c97b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.542721 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.542705    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1c940f3-a6ba-45e5-bb26-2997810c97b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.542963 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.542947    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1c940f3-a6ba-45e5-bb26-2997810c97b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.559742 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.559715    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1c940f3-a6ba-45e5-bb26-2997810c97b6-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.559871 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.559853    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc7td\" (UniqueName: \"kubernetes.io/projected/f1c940f3-a6ba-45e5-bb26-2997810c97b6-kube-api-access-zc7td\") pod \"maas-default-gateway-openshift-default-5bc884dd87-cdt2t\" (UID: \"f1c940f3-a6ba-45e5-bb26-2997810c97b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.659419 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.659396    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:08.966045 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.966023    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"]
Oct 30 19:30:08.967684 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:30:08.967654    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c940f3_a6ba_45e5_bb26_2997810c97b6.slice/crio-9e6f0df5045e9e6e5fd0a34c6b2234fd2896a674513a2b3635b96309f6353e7b WatchSource:0}: Error finding container 9e6f0df5045e9e6e5fd0a34c6b2234fd2896a674513a2b3635b96309f6353e7b: Status 404 returned error can't find the container with id 9e6f0df5045e9e6e5fd0a34c6b2234fd2896a674513a2b3635b96309f6353e7b
Oct 30 19:30:08.969832 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.969797    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:08.969927 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.969865    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:08.969927 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:08.969894    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:09.944790 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:09.944748    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t" event={"ID":"f1c940f3-a6ba-45e5-bb26-2997810c97b6","Type":"ContainerStarted","Data":"8738cc815860113db5a37eeaa1bc47eadc6c5e8ec5d39dc92ffce14b46695e52"}
Oct 30 19:30:09.945247 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:09.944810    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t" event={"ID":"f1c940f3-a6ba-45e5-bb26-2997810c97b6","Type":"ContainerStarted","Data":"9e6f0df5045e9e6e5fd0a34c6b2234fd2896a674513a2b3635b96309f6353e7b"}
Oct 30 19:30:10.025385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:10.025033    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t" podStartSLOduration=2.025014247 podStartE2EDuration="2.025014247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:09.998843329 +0000 UTC m=+563.298105457" watchObservedRunningTime="2026-10-30 19:30:10.025014247 +0000 UTC m=+563.324276378"
Oct 30 19:30:10.660523 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:10.660490    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:10.662142 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:10.662116    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cdt2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 19:30:10.662262 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:10.662163    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t" podUID="f1c940f3-a6ba-45e5-bb26-2997810c97b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 19:30:11.660491 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:11.660457    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cdt2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 19:30:11.660912 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:11.660510    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t" podUID="f1c940f3-a6ba-45e5-bb26-2997810c97b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 19:30:12.660706 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:12.660673    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cdt2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 19:30:12.661091 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:12.660739    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t" podUID="f1c940f3-a6ba-45e5-bb26-2997810c97b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 19:30:13.660612 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:13.660581    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cdt2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 19:30:13.660776 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:13.660648    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t" podUID="f1c940f3-a6ba-45e5-bb26-2997810c97b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 19:30:14.660734 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:14.660692    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cdt2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 19:30:14.660889 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:14.660751    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t" podUID="f1c940f3-a6ba-45e5-bb26-2997810c97b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 19:30:14.922238 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:14.922169    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-744cf976bc-l8stx"
Oct 30 19:30:14.992252 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:14.992217    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz"]
Oct 30 19:30:14.992436 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:14.992402    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz" podUID="935ffff7-de98-4e0b-98ed-79e4016dce59" containerName="manager" containerID="cri-o://ecf575a56c3275172ae8002ebf623c5983ce322a27bdf1752542fab7d6a6861a" gracePeriod=30
Oct 30 19:30:15.231154 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:15.231119    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz"
Oct 30 19:30:15.304833 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:15.304808    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/935ffff7-de98-4e0b-98ed-79e4016dce59-kube-api-access-htcxj\" (UniqueName: \"kubernetes.io/projected/935ffff7-de98-4e0b-98ed-79e4016dce59-kube-api-access-htcxj\") pod \"935ffff7-de98-4e0b-98ed-79e4016dce59\" (UID: \"935ffff7-de98-4e0b-98ed-79e4016dce59\") "
Oct 30 19:30:15.304995 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:15.304896    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/935ffff7-de98-4e0b-98ed-79e4016dce59-metrics-certs\" (UniqueName: \"kubernetes.io/secret/935ffff7-de98-4e0b-98ed-79e4016dce59-metrics-certs\") pod \"935ffff7-de98-4e0b-98ed-79e4016dce59\" (UID: \"935ffff7-de98-4e0b-98ed-79e4016dce59\") "
Oct 30 19:30:15.306941 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:15.306905    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/935ffff7-de98-4e0b-98ed-79e4016dce59-metrics-certs" pod "935ffff7-de98-4e0b-98ed-79e4016dce59" (UID: "935ffff7-de98-4e0b-98ed-79e4016dce59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:15.307046 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:15.306982    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935ffff7-de98-4e0b-98ed-79e4016dce59-kube-api-access-htcxj" pod "935ffff7-de98-4e0b-98ed-79e4016dce59" (UID: "935ffff7-de98-4e0b-98ed-79e4016dce59"). InnerVolumeSpecName "kube-api-access-htcxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:15.406665 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:15.406631    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htcxj\" (UniqueName: \"kubernetes.io/projected/935ffff7-de98-4e0b-98ed-79e4016dce59-kube-api-access-htcxj\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:30:15.406758 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:15.406669    4712 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/935ffff7-de98-4e0b-98ed-79e4016dce59-metrics-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:30:15.660673 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:15.660639    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cdt2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 19:30:15.660822 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:15.660691    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t" podUID="f1c940f3-a6ba-45e5-bb26-2997810c97b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 19:30:15.973212 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:15.973134    4712 generic.go:356] "Generic (PLEG): container finished" podID="935ffff7-de98-4e0b-98ed-79e4016dce59" containerID="ecf575a56c3275172ae8002ebf623c5983ce322a27bdf1752542fab7d6a6861a" exitCode=0
Oct 30 19:30:15.973212 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:15.973174    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz" event={"ID":"935ffff7-de98-4e0b-98ed-79e4016dce59","Type":"ContainerDied","Data":"ecf575a56c3275172ae8002ebf623c5983ce322a27bdf1752542fab7d6a6861a"}
Oct 30 19:30:15.973212 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:15.973197    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz"
Oct 30 19:30:15.973212 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:15.973213    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz" event={"ID":"935ffff7-de98-4e0b-98ed-79e4016dce59","Type":"ContainerDied","Data":"35226de0f22ff46a8d3cda2b9e0ec3a3b80eca7c2eb1ea3b0bcd33d7b09aeb01"}
Oct 30 19:30:15.973734 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:15.973235    4712 scope.go:122] "RemoveContainer" containerID="ecf575a56c3275172ae8002ebf623c5983ce322a27bdf1752542fab7d6a6861a"
Oct 30 19:30:15.981457 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:15.981437    4712 scope.go:122] "RemoveContainer" containerID="ecf575a56c3275172ae8002ebf623c5983ce322a27bdf1752542fab7d6a6861a"
Oct 30 19:30:15.981707 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:30:15.981688    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf575a56c3275172ae8002ebf623c5983ce322a27bdf1752542fab7d6a6861a\": container with ID starting with ecf575a56c3275172ae8002ebf623c5983ce322a27bdf1752542fab7d6a6861a not found: ID does not exist" containerID="ecf575a56c3275172ae8002ebf623c5983ce322a27bdf1752542fab7d6a6861a"
Oct 30 19:30:15.981781 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:15.981716    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf575a56c3275172ae8002ebf623c5983ce322a27bdf1752542fab7d6a6861a"} err="failed to get container status \"ecf575a56c3275172ae8002ebf623c5983ce322a27bdf1752542fab7d6a6861a\": rpc error: code = NotFound desc = could not find container \"ecf575a56c3275172ae8002ebf623c5983ce322a27bdf1752542fab7d6a6861a\": container with ID starting with ecf575a56c3275172ae8002ebf623c5983ce322a27bdf1752542fab7d6a6861a not found: ID does not exist"
Oct 30 19:30:16.023646 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:16.023621    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz"]
Oct 30 19:30:16.035661 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:16.035641    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-6cf7bcbfb6-p8pgz"]
Oct 30 19:30:16.753382 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:16.752601    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:16.852042 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:16.852006    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="935ffff7-de98-4e0b-98ed-79e4016dce59" path="/var/lib/kubelet/pods/935ffff7-de98-4e0b-98ed-79e4016dce59/volumes"
Oct 30 19:30:17.128453 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:17.128425    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:17.128925 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:17.128723    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:17.129405 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:17.129383    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t"
Oct 30 19:30:17.261987 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:17.261959    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:18.896495 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:18.896465    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:19.053821 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:19.053795    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:20.531070 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:20.531035    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:20.691826 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:20.691796    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:30.579160 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:30.579075    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:36.371876 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:30:36.371837    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:30:43.543040 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:43.543008    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:43.978457 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:43.978425    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"]
Oct 30 19:30:43.978793 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:43.978775    4712 cpu_manager.go:414] "containerMap: removing container" podUID="935ffff7-de98-4e0b-98ed-79e4016dce59" containerName="manager"
Oct 30 19:30:43.978793 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:43.978789    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935ffff7-de98-4e0b-98ed-79e4016dce59" containerName="manager"
Oct 30 19:30:43.978985 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:43.978877    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="935ffff7-de98-4e0b-98ed-79e4016dce59" containerName="manager"
Oct 30 19:30:43.982038 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:43.982015    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:43.997016 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:43.996997    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ef9a5a-openshift-default-dockercfg-dtqfz\""
Oct 30 19:30:44.014330 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.014303    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"]
Oct 30 19:30:44.138914 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.138883    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-workload-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.138914 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.138916    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-envoy\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.139118 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.138936    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-workload-certs\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.139118 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.138967    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llgnm\" (UniqueName: \"kubernetes.io/projected/b727df7a-56c7-46fc-974a-38aebc2593c3-kube-api-access-llgnm\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.139118 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.139094    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-credential-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.139235 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.139143    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-token\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.139235 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.139162    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b727df7a-56c7-46fc-974a-38aebc2593c3-istiod-ca-cert\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.139342 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.139293    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-data\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.139421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.139343    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-podinfo\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.240221 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.240144    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-data\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.240221 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.240208    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-podinfo\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.240454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.240249    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-workload-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.240454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.240267    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-envoy\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.240454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.240292    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-workload-certs\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.240454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.240352    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llgnm\" (UniqueName: \"kubernetes.io/projected/b727df7a-56c7-46fc-974a-38aebc2593c3-kube-api-access-llgnm\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.240454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.240431    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-credential-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.240670 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.240488    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-data\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.240670 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.240510    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-token\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.240670 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.240537    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b727df7a-56c7-46fc-974a-38aebc2593c3-istiod-ca-cert\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.240670 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.240568    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-workload-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.241001 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.240971    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-workload-certs\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.241205 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.241185    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b727df7a-56c7-46fc-974a-38aebc2593c3-istiod-ca-cert\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.241266 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.241194    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-credential-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.242820 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.242800    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-podinfo\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.243038 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.243021    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-envoy\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.266733 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.266707    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llgnm\" (UniqueName: \"kubernetes.io/projected/b727df7a-56c7-46fc-974a-38aebc2593c3-kube-api-access-llgnm\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.271356 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.271334    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-token\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.281457 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.281431    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"]
Oct 30 19:30:44.286215 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.286194    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.296801 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.296602    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:44.310377 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.310329    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-544fc8-openshift-default-dockercfg-t64zf\""
Oct 30 19:30:44.312530 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.312508    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"]
Oct 30 19:30:44.443353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.443323    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nbtr\" (UniqueName: \"kubernetes.io/projected/d6ddfa89-9f31-4674-a41e-bcc43761b954-kube-api-access-7nbtr\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.443531 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.443419    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-token\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.443531 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.443457    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-workload-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.443531 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.443482    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-data\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.443531 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.443497    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-envoy\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.443676 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.443598    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-workload-certs\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.443676 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.443637    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-podinfo\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.443787 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.443767    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-credential-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.443844 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.443810    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ddfa89-9f31-4674-a41e-bcc43761b954-istiod-ca-cert\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.544790 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.544714    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-credential-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.544790 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.544757    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ddfa89-9f31-4674-a41e-bcc43761b954-istiod-ca-cert\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.545290 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.544803    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nbtr\" (UniqueName: \"kubernetes.io/projected/d6ddfa89-9f31-4674-a41e-bcc43761b954-kube-api-access-7nbtr\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.545290 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.544870    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-token\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.545290 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.545079    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-workload-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.545290 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.545127    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-data\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.545290 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.545155    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-envoy\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.545290 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.545174    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-credential-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.545290 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.545195    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-workload-certs\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.545754 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.545462    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-data\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.545754 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.545516    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-workload-certs\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.545754 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.545665    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-podinfo\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.545754 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.545707    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ddfa89-9f31-4674-a41e-bcc43761b954-istiod-ca-cert\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.545754 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.545718    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-workload-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.547812 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.547788    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-envoy\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.547907 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.547891    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-podinfo\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.559507 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.559479    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"]
Oct 30 19:30:44.566339 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.565445    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.578324 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.578300    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nbtr\" (UniqueName: \"kubernetes.io/projected/d6ddfa89-9f31-4674-a41e-bcc43761b954-kube-api-access-7nbtr\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.580376 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.580341    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-token\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.583118 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.583092    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f913c6-openshift-default-dockercfg-m5wb4\""
Oct 30 19:30:44.587939 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.587918    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"]
Oct 30 19:30:44.619319 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.619294    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"]
Oct 30 19:30:44.619858 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:30:44.619830    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb727df7a_56c7_46fc_974a_38aebc2593c3.slice/crio-8fe4b2a018f9153a287115040c14625f1d5ef931bde8dd12ff0ca07083dcaad0 WatchSource:0}: Error finding container 8fe4b2a018f9153a287115040c14625f1d5ef931bde8dd12ff0ca07083dcaad0: Status 404 returned error can't find the container with id 8fe4b2a018f9153a287115040c14625f1d5ef931bde8dd12ff0ca07083dcaad0
Oct 30 19:30:44.622279 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.622247    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:44.622345 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.622317    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:44.622423 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.622353    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:44.623243 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.623229    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:44.747079 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.747046    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-envoy\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.747245 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.747092    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-podinfo\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.747245 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.747123    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istiod-ca-cert\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.747245 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.747154    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-token\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.747245 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.747177    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6cz7\" (UniqueName: \"kubernetes.io/projected/3fce9cac-38e0-436f-a4aa-4f82b908e11e-kube-api-access-t6cz7\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.747245 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.747227    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-data\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.747525 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.747251    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-credential-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.747525 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.747295    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-workload-certs\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.747525 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.747339    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-workload-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.847923 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.847846    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-data\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.847923 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.847875    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-credential-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.847923 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.847900    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-workload-certs\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.848210 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.847928    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-workload-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.848210 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.847983    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-envoy\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.848210 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.848074    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-podinfo\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.848210 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.848111    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istiod-ca-cert\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.848210 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.848161    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-token\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.848210 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.848186    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6cz7\" (UniqueName: \"kubernetes.io/projected/3fce9cac-38e0-436f-a4aa-4f82b908e11e-kube-api-access-t6cz7\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.848553 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.848267    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-workload-certs\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.848553 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.848403    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-credential-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.848705 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.848683    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-data\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.848954 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.848934    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-workload-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.849048 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.848999    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istiod-ca-cert\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.850863 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.850836    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-envoy\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.850999 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.850934    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-podinfo\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.881309 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.881286    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6cz7\" (UniqueName: \"kubernetes.io/projected/3fce9cac-38e0-436f-a4aa-4f82b908e11e-kube-api-access-t6cz7\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.887865 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.887842    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-token\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:44.985273 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:30:44.985241    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ddfa89_9f31_4674_a41e_bcc43761b954.slice/crio-fdeb00c97ae0e65d4941020bcec63a8ecc7a03dcd5dfc10c239af029a58a8d54 WatchSource:0}: Error finding container fdeb00c97ae0e65d4941020bcec63a8ecc7a03dcd5dfc10c239af029a58a8d54: Status 404 returned error can't find the container with id fdeb00c97ae0e65d4941020bcec63a8ecc7a03dcd5dfc10c239af029a58a8d54
Oct 30 19:30:44.985690 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.985665    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"]
Oct 30 19:30:44.987858 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.987817    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:44.987940 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.987905    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:44.987979 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:44.987952    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:45.094599 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.094043    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd" event={"ID":"d6ddfa89-9f31-4674-a41e-bcc43761b954","Type":"ContainerStarted","Data":"1365a70761faa306c574957470df665ddeafe740d8d77e01d5f213c474d5f470"}
Oct 30 19:30:45.094599 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.094117    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd" event={"ID":"d6ddfa89-9f31-4674-a41e-bcc43761b954","Type":"ContainerStarted","Data":"fdeb00c97ae0e65d4941020bcec63a8ecc7a03dcd5dfc10c239af029a58a8d54"}
Oct 30 19:30:45.096405 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.096287    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc" event={"ID":"b727df7a-56c7-46fc-974a-38aebc2593c3","Type":"ContainerStarted","Data":"9b53887b9a54a31204a0e16b19ff4331e4dd29b74539b697a6bf066b43cfc056"}
Oct 30 19:30:45.096405 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.096348    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc" event={"ID":"b727df7a-56c7-46fc-974a-38aebc2593c3","Type":"ContainerStarted","Data":"8fe4b2a018f9153a287115040c14625f1d5ef931bde8dd12ff0ca07083dcaad0"}
Oct 30 19:30:45.099843 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.099780    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"]
Oct 30 19:30:45.103253 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.103234    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.119514 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.119496    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-4b4c34-openshift-default-dockercfg-w76ld\""
Oct 30 19:30:45.137145 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.137123    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"]
Oct 30 19:30:45.162442 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.162400    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd" podStartSLOduration=1.162389421 podStartE2EDuration="1.162389421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:45.161496844 +0000 UTC m=+598.460758972" watchObservedRunningTime="2026-10-30 19:30:45.162389421 +0000 UTC m=+598.461651544"
Oct 30 19:30:45.181327 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.181300    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:45.253511 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.253409    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-envoy\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.253511 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.253502    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-data\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.253761 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.253595    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-workload-certs\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.253761 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.253751    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzcbq\" (UniqueName: \"kubernetes.io/projected/045af806-47d7-41b6-9cd7-ab0b9c851f3c-kube-api-access-hzcbq\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.253900 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.253792    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istiod-ca-cert\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.253900 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.253845    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-workload-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.253900 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.253880    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-credential-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.254053 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.253914    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-podinfo\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.254053 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.253944    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-token\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.257492 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.256927    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc" podStartSLOduration=2.25691448 podStartE2EDuration="2.25691448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:45.255412498 +0000 UTC m=+598.554674627" watchObservedRunningTime="2026-10-30 19:30:45.25691448 +0000 UTC m=+598.556176610"
Oct 30 19:30:45.299573 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.299546    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:45.304123 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.304101    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:45.355476 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.355405    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-workload-certs\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.355584 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.355500    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzcbq\" (UniqueName: \"kubernetes.io/projected/045af806-47d7-41b6-9cd7-ab0b9c851f3c-kube-api-access-hzcbq\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.355584 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.355537    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istiod-ca-cert\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.355584 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.355579    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-workload-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.355751 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.355611    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-credential-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.355751 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.355643    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-podinfo\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.355751 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.355670    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-token\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.355751 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.355703    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-envoy\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.355939 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.355761    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-data\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.355939 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.355909    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-workload-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.356048 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.356030    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-credential-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.356149 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.356129    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-data\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.356209 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.356134    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-workload-certs\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.356290 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.356274    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istiod-ca-cert\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.358279 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.358255    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-podinfo\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.358279 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.358267    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-envoy\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.388244 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.388212    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-token\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.388724 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.388689    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzcbq\" (UniqueName: \"kubernetes.io/projected/045af806-47d7-41b6-9cd7-ab0b9c851f3c-kube-api-access-hzcbq\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.416833 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.416802    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:45.490702 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.490620    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"]
Oct 30 19:30:45.492628 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:30:45.492595    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fce9cac_38e0_436f_a4aa_4f82b908e11e.slice/crio-c2acd6360ee2b6eb5983b181dedaaf78673a042f903f99f8e7c1da238e6e49f6 WatchSource:0}: Error finding container c2acd6360ee2b6eb5983b181dedaaf78673a042f903f99f8e7c1da238e6e49f6: Status 404 returned error can't find the container with id c2acd6360ee2b6eb5983b181dedaaf78673a042f903f99f8e7c1da238e6e49f6
Oct 30 19:30:45.495044 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.494999    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:45.495125 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.495091    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:45.495187 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.495139    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:45.534287 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.534264    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:45.623682 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.623646    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:45.715348 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.715328    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"]
Oct 30 19:30:45.717881 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:30:45.717852    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045af806_47d7_41b6_9cd7_ab0b9c851f3c.slice/crio-abe545488a61409c43591a5a267f97032ae3c213a87063e5280b0700bc8f4d4d WatchSource:0}: Error finding container abe545488a61409c43591a5a267f97032ae3c213a87063e5280b0700bc8f4d4d: Status 404 returned error can't find the container with id abe545488a61409c43591a5a267f97032ae3c213a87063e5280b0700bc8f4d4d
Oct 30 19:30:45.720342 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.720302    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:45.720460 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.720404    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:45.720460 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:45.720448    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:46.102386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:46.102286    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp" event={"ID":"3fce9cac-38e0-436f-a4aa-4f82b908e11e","Type":"ContainerStarted","Data":"a2d80302043fb98ff8f87b011a770cfa15237d7ea139100c7a7ef665859d1421"}
Oct 30 19:30:46.102386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:46.102350    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp" event={"ID":"3fce9cac-38e0-436f-a4aa-4f82b908e11e","Type":"ContainerStarted","Data":"c2acd6360ee2b6eb5983b181dedaaf78673a042f903f99f8e7c1da238e6e49f6"}
Oct 30 19:30:46.103999 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:46.103970    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst" event={"ID":"045af806-47d7-41b6-9cd7-ab0b9c851f3c","Type":"ContainerStarted","Data":"d4d13207653fbc31f850ff0f17e0e35840f4488515bdad29a82d1b7357365a07"}
Oct 30 19:30:46.104139 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:46.104016    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst" event={"ID":"045af806-47d7-41b6-9cd7-ab0b9c851f3c","Type":"ContainerStarted","Data":"abe545488a61409c43591a5a267f97032ae3c213a87063e5280b0700bc8f4d4d"}
Oct 30 19:30:46.104406 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:46.104386    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:46.105118 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:46.105101    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:30:46.154332 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:46.154286    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp" podStartSLOduration=2.154273073 podStartE2EDuration="2.154273073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:46.153687752 +0000 UTC m=+599.452949882" watchObservedRunningTime="2026-10-30 19:30:46.154273073 +0000 UTC m=+599.453535204"
Oct 30 19:30:46.181736 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:46.181711    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:46.186595 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:46.186566    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:46.199331 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:46.199289    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst" podStartSLOduration=1.199278471 podStartE2EDuration="1.199278471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:46.197780933 +0000 UTC m=+599.497043062" watchObservedRunningTime="2026-10-30 19:30:46.199278471 +0000 UTC m=+599.498540636"
Oct 30 19:30:46.418039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:46.418001    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:46.422857 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:46.422831    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:46.628903 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:46.628872    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:46.889609 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:46.889582    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:30:46.890614 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:46.890588    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:30:47.108225 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:47.108183    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:47.108425 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:47.108274    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:47.108765 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:47.108737    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:47.109023 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:47.109002    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:30:47.109199 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:47.109180    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:30:47.109638 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:47.109621    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:30:47.863775 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:47.863743    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:48.044023 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:48.043977    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:48.226606 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:48.226569    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:48.399739 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:48.399712    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:48.867677 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:48.867647    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:49.078821 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:49.078788    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:49.267513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:49.267444    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:49.457223 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:49.457195    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:49.633769 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:49.633737    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:50.177322 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.177294    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:50.242627 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.242598    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c"]
Oct 30 19:30:50.247970 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.247940    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c"
Oct 30 19:30:50.265959 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.265935    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c"]
Oct 30 19:30:50.298908 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.298880    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-d797d2-dockercfg-dlmv7\""
Oct 30 19:30:50.299452 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.299433    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19408be-2ea5-4445-99a5-9094c497b95f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c\" (UID: \"d19408be-2ea5-4445-99a5-9094c497b95f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c"
Oct 30 19:30:50.299576 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.299561    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49hjj\" (UniqueName: \"kubernetes.io/projected/d19408be-2ea5-4445-99a5-9094c497b95f-kube-api-access-49hjj\") pod \"payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c\" (UID: \"d19408be-2ea5-4445-99a5-9094c497b95f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c"
Oct 30 19:30:50.332869 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.332845    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:50.351997 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.351980    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-d797d2\""
Oct 30 19:30:50.400397 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.400346    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19408be-2ea5-4445-99a5-9094c497b95f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c\" (UID: \"d19408be-2ea5-4445-99a5-9094c497b95f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c"
Oct 30 19:30:50.400571 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.400463    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49hjj\" (UniqueName: \"kubernetes.io/projected/d19408be-2ea5-4445-99a5-9094c497b95f-kube-api-access-49hjj\") pod \"payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c\" (UID: \"d19408be-2ea5-4445-99a5-9094c497b95f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c"
Oct 30 19:30:50.401127 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.401102    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19408be-2ea5-4445-99a5-9094c497b95f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c\" (UID: \"d19408be-2ea5-4445-99a5-9094c497b95f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c"
Oct 30 19:30:50.419919 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.419890    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49hjj\" (UniqueName: \"kubernetes.io/projected/d19408be-2ea5-4445-99a5-9094c497b95f-kube-api-access-49hjj\") pod \"payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c\" (UID: \"d19408be-2ea5-4445-99a5-9094c497b95f\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c"
Oct 30 19:30:50.569659 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.569595    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c"
Oct 30 19:30:50.844929 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.844854    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"]
Oct 30 19:30:50.849604 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.849579    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:30:50.867481 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.867453    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-ef9a5a-serving-cert\""
Oct 30 19:30:50.870536 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.870479    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"]
Oct 30 19:30:50.873397 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:30:50.873334    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19408be_2ea5_4445_99a5_9094c497b95f.slice/crio-2dda31542479d740830f375741e5fa77b27cc061b613be7a2772bcc1cbb2d49c WatchSource:0}: Error finding container 2dda31542479d740830f375741e5fa77b27cc061b613be7a2772bcc1cbb2d49c: Status 404 returned error can't find the container with id 2dda31542479d740830f375741e5fa77b27cc061b613be7a2772bcc1cbb2d49c
Oct 30 19:30:50.882166 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.882142    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c"]
Oct 30 19:30:50.906321 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.906297    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jhl5\" (UniqueName: \"kubernetes.io/projected/b3cd29f0-6462-40b1-a5a0-30deec5e9321-kube-api-access-9jhl5\") pod \"maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq\" (UID: \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:30:50.906446 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.906348    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cd29f0-6462-40b1-a5a0-30deec5e9321-maas-api-tls\") pod \"maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq\" (UID: \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:30:50.906548 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.906526    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cd29f0-6462-40b1-a5a0-30deec5e9321-metrics-certs\") pod \"maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq\" (UID: \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:30:50.906598 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:50.906569    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cd29f0-6462-40b1-a5a0-30deec5e9321-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq\" (UID: \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:30:51.007614 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.007583    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cd29f0-6462-40b1-a5a0-30deec5e9321-metrics-certs\") pod \"maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq\" (UID: \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:30:51.007738 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.007621    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cd29f0-6462-40b1-a5a0-30deec5e9321-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq\" (UID: \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:30:51.007738 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.007699    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jhl5\" (UniqueName: \"kubernetes.io/projected/b3cd29f0-6462-40b1-a5a0-30deec5e9321-kube-api-access-9jhl5\") pod \"maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq\" (UID: \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:30:51.007854 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.007749    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cd29f0-6462-40b1-a5a0-30deec5e9321-maas-api-tls\") pod \"maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq\" (UID: \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:30:51.007915 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:30:51.007893    4712 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-serving-cert: secret "maas-api-e2e-worker-w6-ef9a5a-serving-cert" not found
Oct 30 19:30:51.007983 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:30:51.007970    4712 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3cd29f0-6462-40b1-a5a0-30deec5e9321-maas-api-tls podName:b3cd29f0-6462-40b1-a5a0-30deec5e9321 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:51.507947606 +0000 UTC m=+604.807209730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b3cd29f0-6462-40b1-a5a0-30deec5e9321-maas-api-tls") pod "maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq" (UID: "b3cd29f0-6462-40b1-a5a0-30deec5e9321") : secret "maas-api-e2e-worker-w6-ef9a5a-serving-cert" not found
Oct 30 19:30:51.008378 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.008343    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cd29f0-6462-40b1-a5a0-30deec5e9321-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq\" (UID: \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:30:51.009888 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.009870    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cd29f0-6462-40b1-a5a0-30deec5e9321-metrics-certs\") pod \"maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq\" (UID: \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:30:51.033936 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.033853    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jhl5\" (UniqueName: \"kubernetes.io/projected/b3cd29f0-6462-40b1-a5a0-30deec5e9321-kube-api-access-9jhl5\") pod \"maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq\" (UID: \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:30:51.087744 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.087715    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:51.127907 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.127860    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c" event={"ID":"d19408be-2ea5-4445-99a5-9094c497b95f","Type":"ContainerStarted","Data":"6122c058157a02ff805345e35e90d1c10ab70a725c5d40017f7d6d400815d43d"}
Oct 30 19:30:51.128053 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.127936    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c" event={"ID":"d19408be-2ea5-4445-99a5-9094c497b95f","Type":"ContainerStarted","Data":"2dda31542479d740830f375741e5fa77b27cc061b613be7a2772bcc1cbb2d49c"}
Oct 30 19:30:51.128053 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.127966    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c"
Oct 30 19:30:51.171853 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.171793    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c" podStartSLOduration=1.171774302 podStartE2EDuration="1.171774302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:51.168809117 +0000 UTC m=+604.468071245" watchObservedRunningTime="2026-10-30 19:30:51.171774302 +0000 UTC m=+604.471036431"
Oct 30 19:30:51.237987 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.237955    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:51.395014 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.394929    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:51.512807 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.512779    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cd29f0-6462-40b1-a5a0-30deec5e9321-maas-api-tls\") pod \"maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq\" (UID: \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:30:51.514906 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.514880    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cd29f0-6462-40b1-a5a0-30deec5e9321-maas-api-tls\") pod \"maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq\" (UID: \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:30:51.763141 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.763054    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:30:51.772511 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.772481    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:51.931251 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:51.931214    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:52.088535 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.088506    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"]
Oct 30 19:30:52.089841 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:30:52.089801    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3cd29f0_6462_40b1_a5a0_30deec5e9321.slice/crio-1ef67f20943594828f6a7bc330b314abf28028603e7e880fa792f7ab6aa659d3 WatchSource:0}: Error finding container 1ef67f20943594828f6a7bc330b314abf28028603e7e880fa792f7ab6aa659d3: Status 404 returned error can't find the container with id 1ef67f20943594828f6a7bc330b314abf28028603e7e880fa792f7ab6aa659d3
Oct 30 19:30:52.133996 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.133966    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq" event={"ID":"b3cd29f0-6462-40b1-a5a0-30deec5e9321","Type":"ContainerStarted","Data":"1ef67f20943594828f6a7bc330b314abf28028603e7e880fa792f7ab6aa659d3"}
Oct 30 19:30:52.584545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.584518    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:52.624621 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.624591    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"]
Oct 30 19:30:52.628530 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.628511    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"
Oct 30 19:30:52.648410 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.646879    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-544fc8-serving-cert\""
Oct 30 19:30:52.651512 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.651456    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"]
Oct 30 19:30:52.727810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.727741    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa47487f-ee16-4984-9091-1f277d564af2-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2\" (UID: \"aa47487f-ee16-4984-9091-1f277d564af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"
Oct 30 19:30:52.727810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.727794    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa47487f-ee16-4984-9091-1f277d564af2-maas-api-tls\") pod \"maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2\" (UID: \"aa47487f-ee16-4984-9091-1f277d564af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"
Oct 30 19:30:52.728006 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.727968    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa47487f-ee16-4984-9091-1f277d564af2-metrics-certs\") pod \"maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2\" (UID: \"aa47487f-ee16-4984-9091-1f277d564af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"
Oct 30 19:30:52.728065 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.728022    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcbbj\" (UniqueName: \"kubernetes.io/projected/aa47487f-ee16-4984-9091-1f277d564af2-kube-api-access-dcbbj\") pod \"maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2\" (UID: \"aa47487f-ee16-4984-9091-1f277d564af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"
Oct 30 19:30:52.749958 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.749932    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:52.828839 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.828809    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa47487f-ee16-4984-9091-1f277d564af2-metrics-certs\") pod \"maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2\" (UID: \"aa47487f-ee16-4984-9091-1f277d564af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"
Oct 30 19:30:52.828992 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.828870    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcbbj\" (UniqueName: \"kubernetes.io/projected/aa47487f-ee16-4984-9091-1f277d564af2-kube-api-access-dcbbj\") pod \"maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2\" (UID: \"aa47487f-ee16-4984-9091-1f277d564af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"
Oct 30 19:30:52.828992 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.828972    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa47487f-ee16-4984-9091-1f277d564af2-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2\" (UID: \"aa47487f-ee16-4984-9091-1f277d564af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"
Oct 30 19:30:52.829112 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.829030    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa47487f-ee16-4984-9091-1f277d564af2-maas-api-tls\") pod \"maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2\" (UID: \"aa47487f-ee16-4984-9091-1f277d564af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"
Oct 30 19:30:52.830150 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.830122    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa47487f-ee16-4984-9091-1f277d564af2-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2\" (UID: \"aa47487f-ee16-4984-9091-1f277d564af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"
Oct 30 19:30:52.831222 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.831197    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa47487f-ee16-4984-9091-1f277d564af2-metrics-certs\") pod \"maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2\" (UID: \"aa47487f-ee16-4984-9091-1f277d564af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"
Oct 30 19:30:52.831487 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.831470    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa47487f-ee16-4984-9091-1f277d564af2-maas-api-tls\") pod \"maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2\" (UID: \"aa47487f-ee16-4984-9091-1f277d564af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"
Oct 30 19:30:52.848676 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.848652    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcbbj\" (UniqueName: \"kubernetes.io/projected/aa47487f-ee16-4984-9091-1f277d564af2-kube-api-access-dcbbj\") pod \"maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2\" (UID: \"aa47487f-ee16-4984-9091-1f277d564af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"
Oct 30 19:30:52.899392 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.889590    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:52.943401 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:52.943338    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"
Oct 30 19:30:53.140089 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:53.140044    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq" event={"ID":"b3cd29f0-6462-40b1-a5a0-30deec5e9321","Type":"ContainerStarted","Data":"c56e87437a99acd7ea3b523aeaa619c1a690cc570f243443c4f91ade22ef282f"}
Oct 30 19:30:53.140250 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:53.140118    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:30:53.180455 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:53.180408    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq" podStartSLOduration=2.837868683 podStartE2EDuration="3.180394314s" totalImagesPullingTime="342.525631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:53.179193564 +0000 UTC m=+606.478455676" watchObservedRunningTime="2026-10-30 19:30:53.180394314 +0000 UTC m=+606.479656442"
Oct 30 19:30:53.239397 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:53.239064    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"]
Oct 30 19:30:53.240755 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:30:53.240674    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa47487f_ee16_4984_9091_1f277d564af2.slice/crio-ecf6aae3fe9eea240cde0ea98641112a5ab1c9f23d8bb676a2672b485d5ca347 WatchSource:0}: Error finding container ecf6aae3fe9eea240cde0ea98641112a5ab1c9f23d8bb676a2672b485d5ca347: Status 404 returned error can't find the container with id ecf6aae3fe9eea240cde0ea98641112a5ab1c9f23d8bb676a2672b485d5ca347
Oct 30 19:30:53.423561 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:53.423530    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:53.591400 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:53.591296    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:54.145931 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:54.145889    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2" event={"ID":"aa47487f-ee16-4984-9091-1f277d564af2","Type":"ContainerStarted","Data":"d416c951ebbb5acb038fe7f0f5eb253a3404a3a81206c77514b9c8b8fd56f300"}
Oct 30 19:30:54.146129 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:54.145969    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2" event={"ID":"aa47487f-ee16-4984-9091-1f277d564af2","Type":"ContainerStarted","Data":"ecf6aae3fe9eea240cde0ea98641112a5ab1c9f23d8bb676a2672b485d5ca347"}
Oct 30 19:30:54.146212 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:54.146134    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"
Oct 30 19:30:54.185445 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:54.185401    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2" podStartSLOduration=1.773058858 podStartE2EDuration="2.185389746s" totalImagesPullingTime="412.330888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:54.185272294 +0000 UTC m=+607.484534422" watchObservedRunningTime="2026-10-30 19:30:54.185389746 +0000 UTC m=+607.484651874"
Oct 30 19:30:54.370019 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:54.369985    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:54.522290 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:54.522214    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:54.908071 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:54.908042    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:55.002115 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.002085    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr"]
Oct 30 19:30:55.005601 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.005586    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr"
Oct 30 19:30:55.024280 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.024255    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr"]
Oct 30 19:30:55.034751 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.034731    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-f913c6\""
Oct 30 19:30:55.034852 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.034836    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-f913c6-dockercfg-72c76\""
Oct 30 19:30:55.053550 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.053522    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/245c3fb3-0b83-4d71-be3c-81d6cafb639c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr\" (UID: \"245c3fb3-0b83-4d71-be3c-81d6cafb639c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr"
Oct 30 19:30:55.053672 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.053650    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g2hz\" (UniqueName: \"kubernetes.io/projected/245c3fb3-0b83-4d71-be3c-81d6cafb639c-kube-api-access-4g2hz\") pod \"payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr\" (UID: \"245c3fb3-0b83-4d71-be3c-81d6cafb639c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr"
Oct 30 19:30:55.068843 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.068821    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:55.119884 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.119860    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59"]
Oct 30 19:30:55.123294 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.123279    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59"
Oct 30 19:30:55.154186 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.154162    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/245c3fb3-0b83-4d71-be3c-81d6cafb639c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr\" (UID: \"245c3fb3-0b83-4d71-be3c-81d6cafb639c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr"
Oct 30 19:30:55.154291 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.154194    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd990c24-235b-4f45-92a1-da3e7c443153-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59\" (UID: \"fd990c24-235b-4f45-92a1-da3e7c443153\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59"
Oct 30 19:30:55.154291 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.154218    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcrkn\" (UniqueName: \"kubernetes.io/projected/fd990c24-235b-4f45-92a1-da3e7c443153-kube-api-access-dcrkn\") pod \"payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59\" (UID: \"fd990c24-235b-4f45-92a1-da3e7c443153\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59"
Oct 30 19:30:55.154291 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.154259    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g2hz\" (UniqueName: \"kubernetes.io/projected/245c3fb3-0b83-4d71-be3c-81d6cafb639c-kube-api-access-4g2hz\") pod \"payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr\" (UID: \"245c3fb3-0b83-4d71-be3c-81d6cafb639c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr"
Oct 30 19:30:55.154879 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.154859    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/245c3fb3-0b83-4d71-be3c-81d6cafb639c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr\" (UID: \"245c3fb3-0b83-4d71-be3c-81d6cafb639c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr"
Oct 30 19:30:55.174598 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.174550    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59"]
Oct 30 19:30:55.180221 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.180199    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g2hz\" (UniqueName: \"kubernetes.io/projected/245c3fb3-0b83-4d71-be3c-81d6cafb639c-kube-api-access-4g2hz\") pod \"payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr\" (UID: \"245c3fb3-0b83-4d71-be3c-81d6cafb639c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr"
Oct 30 19:30:55.255277 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.255240    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd990c24-235b-4f45-92a1-da3e7c443153-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59\" (UID: \"fd990c24-235b-4f45-92a1-da3e7c443153\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59"
Oct 30 19:30:55.255277 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.255294    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcrkn\" (UniqueName: \"kubernetes.io/projected/fd990c24-235b-4f45-92a1-da3e7c443153-kube-api-access-dcrkn\") pod \"payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59\" (UID: \"fd990c24-235b-4f45-92a1-da3e7c443153\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59"
Oct 30 19:30:55.255277 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.257329    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd990c24-235b-4f45-92a1-da3e7c443153-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59\" (UID: \"fd990c24-235b-4f45-92a1-da3e7c443153\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59"
Oct 30 19:30:55.295920 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.295827    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcrkn\" (UniqueName: \"kubernetes.io/projected/fd990c24-235b-4f45-92a1-da3e7c443153-kube-api-access-dcrkn\") pod \"payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59\" (UID: \"fd990c24-235b-4f45-92a1-da3e7c443153\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59"
Oct 30 19:30:55.318450 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.318417    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr"
Oct 30 19:30:55.435820 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.435747    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59"
Oct 30 19:30:55.612780 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.612754    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr"]
Oct 30 19:30:55.613054 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:30:55.613037    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245c3fb3_0b83_4d71_be3c_81d6cafb639c.slice/crio-4983e00ca5220ccd2eb84e78e25dc5dd82a357e6c05dc54080422b2c14fea5e5 WatchSource:0}: Error finding container 4983e00ca5220ccd2eb84e78e25dc5dd82a357e6c05dc54080422b2c14fea5e5: Status 404 returned error can't find the container with id 4983e00ca5220ccd2eb84e78e25dc5dd82a357e6c05dc54080422b2c14fea5e5
Oct 30 19:30:55.753466 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.753298    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59"]
Oct 30 19:30:55.762272 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:30:55.762240    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd990c24_235b_4f45_92a1_da3e7c443153.slice/crio-f50b33c317382b3da5ab25863d95fdac0bf9bd5121ea0bb4450eb64340c3f1fa WatchSource:0}: Error finding container f50b33c317382b3da5ab25863d95fdac0bf9bd5121ea0bb4450eb64340c3f1fa: Status 404 returned error can't find the container with id f50b33c317382b3da5ab25863d95fdac0bf9bd5121ea0bb4450eb64340c3f1fa
Oct 30 19:30:55.781671 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.781647    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:55.840041 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.840017    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"]
Oct 30 19:30:55.843643 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.843626    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"
Oct 30 19:30:55.858338 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.858320    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-3735b2-serving-cert\""
Oct 30 19:30:55.868548 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.868524    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"]
Oct 30 19:30:55.954877 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.954846    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:55.963177 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.963149    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn\" (UID: \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"
Oct 30 19:30:55.963319 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.963195    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r92v8\" (UniqueName: \"kubernetes.io/projected/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-kube-api-access-r92v8\") pod \"maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn\" (UID: \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"
Oct 30 19:30:55.963319 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.963289    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-maas-api-tls\") pod \"maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn\" (UID: \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"
Oct 30 19:30:55.963430 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:55.963354    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-metrics-certs\") pod \"maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn\" (UID: \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"
Oct 30 19:30:56.064219 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.064146    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn\" (UID: \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"
Oct 30 19:30:56.064219 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.064182    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r92v8\" (UniqueName: \"kubernetes.io/projected/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-kube-api-access-r92v8\") pod \"maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn\" (UID: \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"
Oct 30 19:30:56.064467 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.064316    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-maas-api-tls\") pod \"maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn\" (UID: \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"
Oct 30 19:30:56.064467 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.064419    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-metrics-certs\") pod \"maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn\" (UID: \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"
Oct 30 19:30:56.064990 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.064956    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn\" (UID: \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"
Oct 30 19:30:56.067039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.067011    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-metrics-certs\") pod \"maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn\" (UID: \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"
Oct 30 19:30:56.067178 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.067017    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-maas-api-tls\") pod \"maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn\" (UID: \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"
Oct 30 19:30:56.083919 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.083890    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r92v8\" (UniqueName: \"kubernetes.io/projected/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-kube-api-access-r92v8\") pod \"maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn\" (UID: \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"
Oct 30 19:30:56.123384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.123180    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:56.157885 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.157587    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"
Oct 30 19:30:56.169763 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.169723    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr" event={"ID":"245c3fb3-0b83-4d71-be3c-81d6cafb639c","Type":"ContainerStarted","Data":"4aab104c581d69119fa60e244a86bebc65547f7a455fbf9663d95a6b87749a2f"}
Oct 30 19:30:56.170047 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.170024    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr" event={"ID":"245c3fb3-0b83-4d71-be3c-81d6cafb639c","Type":"ContainerStarted","Data":"4983e00ca5220ccd2eb84e78e25dc5dd82a357e6c05dc54080422b2c14fea5e5"}
Oct 30 19:30:56.171079 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.171057    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr"
Oct 30 19:30:56.174421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.174347    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59" event={"ID":"fd990c24-235b-4f45-92a1-da3e7c443153","Type":"ContainerStarted","Data":"d0f7dcf6c92940f10ae3c0c4ec262df3e7ec3b818eed42d26f606a0dff54cf33"}
Oct 30 19:30:56.174537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.174449    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59" event={"ID":"fd990c24-235b-4f45-92a1-da3e7c443153","Type":"ContainerStarted","Data":"f50b33c317382b3da5ab25863d95fdac0bf9bd5121ea0bb4450eb64340c3f1fa"}
Oct 30 19:30:56.175095 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.175075    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59"
Oct 30 19:30:56.278956 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.278797    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr" podStartSLOduration=2.278782865 podStartE2EDuration="2.278782865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:56.230406332 +0000 UTC m=+609.529668462" watchObservedRunningTime="2026-10-30 19:30:56.278782865 +0000 UTC m=+609.578044994"
Oct 30 19:30:56.278956 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.278911    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59" podStartSLOduration=1.278904301 podStartE2EDuration="1.278904301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:56.275810174 +0000 UTC m=+609.575072297" watchObservedRunningTime="2026-10-30 19:30:56.278904301 +0000 UTC m=+609.578166432"
Oct 30 19:30:56.559425 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:56.559393    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"]
Oct 30 19:30:57.182355 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:57.182232    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn" event={"ID":"27c8e215-0d99-44f8-88cd-8c5fe3b964b1","Type":"ContainerStarted","Data":"78eec3eddcb29273daf501ee57350efeecc3a265b02da12aa6745bdefe8c382d"}
Oct 30 19:30:57.182355 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:57.182315    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn" event={"ID":"27c8e215-0d99-44f8-88cd-8c5fe3b964b1","Type":"ContainerStarted","Data":"c3e5bd11fc11152954d27d33d88f3f096f83fa4d37b2ade4188347dbb03caae4"}
Oct 30 19:30:57.182893 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:57.182422    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"
Oct 30 19:30:57.227388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:57.224315    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn" podStartSLOduration=1.9016708599999999 podStartE2EDuration="2.224296394s" totalImagesPullingTime="322.625534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:57.221771616 +0000 UTC m=+610.521033759" watchObservedRunningTime="2026-10-30 19:30:57.224296394 +0000 UTC m=+610.523558523"
Oct 30 19:30:57.484812 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:57.484731    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:57.632930 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:57.632853    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:57.800898 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:57.800800    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:30:59.160264 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:59.160220    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:30:59.197708 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:30:59.197661    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:00.159000 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:00.158965    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"
Oct 30 19:31:00.281388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:00.277708    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:00.437435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:00.436583    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:00.765408 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:00.765295    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:00.921527 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:00.920655    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:01.124028 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:01.123995    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:02.136909 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.136882    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c"
Oct 30 19:31:02.451845 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.451760    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"]
Oct 30 19:31:02.460747 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.460495    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"
Oct 30 19:31:02.482229 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.478720    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"]
Oct 30 19:31:02.490196 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.487415    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-dd23d1-serving-cert\""
Oct 30 19:31:02.490196 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.487885    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96d155d9-0c83-46ed-b34a-81de99b24aa1-metrics-certs\") pod \"maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g\" (UID: \"96d155d9-0c83-46ed-b34a-81de99b24aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"
Oct 30 19:31:02.490196 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.487950    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96d155d9-0c83-46ed-b34a-81de99b24aa1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g\" (UID: \"96d155d9-0c83-46ed-b34a-81de99b24aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"
Oct 30 19:31:02.490196 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.488022    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d155d9-0c83-46ed-b34a-81de99b24aa1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g\" (UID: \"96d155d9-0c83-46ed-b34a-81de99b24aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"
Oct 30 19:31:02.490196 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.488091    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmjvx\" (UniqueName: \"kubernetes.io/projected/96d155d9-0c83-46ed-b34a-81de99b24aa1-kube-api-access-vmjvx\") pod \"maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g\" (UID: \"96d155d9-0c83-46ed-b34a-81de99b24aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"
Oct 30 19:31:02.589370 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.589318    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d155d9-0c83-46ed-b34a-81de99b24aa1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g\" (UID: \"96d155d9-0c83-46ed-b34a-81de99b24aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"
Oct 30 19:31:02.589548 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.589415    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmjvx\" (UniqueName: \"kubernetes.io/projected/96d155d9-0c83-46ed-b34a-81de99b24aa1-kube-api-access-vmjvx\") pod \"maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g\" (UID: \"96d155d9-0c83-46ed-b34a-81de99b24aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"
Oct 30 19:31:02.589548 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.589480    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96d155d9-0c83-46ed-b34a-81de99b24aa1-metrics-certs\") pod \"maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g\" (UID: \"96d155d9-0c83-46ed-b34a-81de99b24aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"
Oct 30 19:31:02.589548 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.589527    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96d155d9-0c83-46ed-b34a-81de99b24aa1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g\" (UID: \"96d155d9-0c83-46ed-b34a-81de99b24aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"
Oct 30 19:31:02.590257 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.590220    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d155d9-0c83-46ed-b34a-81de99b24aa1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g\" (UID: \"96d155d9-0c83-46ed-b34a-81de99b24aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"
Oct 30 19:31:02.592124 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.592098    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96d155d9-0c83-46ed-b34a-81de99b24aa1-metrics-certs\") pod \"maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g\" (UID: \"96d155d9-0c83-46ed-b34a-81de99b24aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"
Oct 30 19:31:02.592553 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.592532    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96d155d9-0c83-46ed-b34a-81de99b24aa1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g\" (UID: \"96d155d9-0c83-46ed-b34a-81de99b24aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"
Oct 30 19:31:02.594513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.594488    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:02.656102 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.656058    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmjvx\" (UniqueName: \"kubernetes.io/projected/96d155d9-0c83-46ed-b34a-81de99b24aa1-kube-api-access-vmjvx\") pod \"maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g\" (UID: \"96d155d9-0c83-46ed-b34a-81de99b24aa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"
Oct 30 19:31:02.776602 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:02.776522    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"
Oct 30 19:31:03.151498 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:03.139793    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"]
Oct 30 19:31:03.178591 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:03.178478    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:31:03.178591 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:03.178544    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd" podUID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:31:03.195869 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:03.195772    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"
Oct 30 19:31:03.236024 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:03.235936    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g" event={"ID":"96d155d9-0c83-46ed-b34a-81de99b24aa1","Type":"ContainerStarted","Data":"f8020a4eca79dbf4663aeccb53c4b43e7a37a9d5c3b2202d6dbc4df061ad2b17"}
Oct 30 19:31:03.766382 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:03.766203    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:04.163445 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:04.163408    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:04.267998 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:04.267961    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g" event={"ID":"96d155d9-0c83-46ed-b34a-81de99b24aa1","Type":"ContainerStarted","Data":"dacd04f615d5fa99a344fc29a071d618f4c3d1b2868c55308a355407252d9bd1"}
Oct 30 19:31:04.268696 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:04.268673    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"
Oct 30 19:31:04.326414 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:04.315609    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g" podStartSLOduration=1.893215139 podStartE2EDuration="2.315589587s" totalImagesPullingTime="422.374448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:04.312740908 +0000 UTC m=+617.612003038" watchObservedRunningTime="2026-10-30 19:31:04.315589587 +0000 UTC m=+617.614851717"
Oct 30 19:31:04.567731 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:04.567618    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:04.726294 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:04.726258    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:06.094665 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:06.094625    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:06.386352 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:31:06.386280    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:31:08.039638 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.039602    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:08.193619 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.192470    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr"
Oct 30 19:31:08.193900 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.193873    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59"
Oct 30 19:31:08.198052 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.198027    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:08.294389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.293539    4712 generic.go:356] "Generic (PLEG): container finished" podID="b727df7a-56c7-46fc-974a-38aebc2593c3" containerID="9b53887b9a54a31204a0e16b19ff4331e4dd29b74539b697a6bf066b43cfc056" exitCode=0
Oct 30 19:31:08.294389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.293670    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc" event={"ID":"b727df7a-56c7-46fc-974a-38aebc2593c3","Type":"ContainerDied","Data":"9b53887b9a54a31204a0e16b19ff4331e4dd29b74539b697a6bf066b43cfc056"}
Oct 30 19:31:08.294389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.294113    4712 scope.go:122] "RemoveContainer" containerID="9b53887b9a54a31204a0e16b19ff4331e4dd29b74539b697a6bf066b43cfc056"
Oct 30 19:31:08.296034 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.295997    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:31:08.296135 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.296087    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:31:08.296197 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.296133    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:31:08.302870 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.302847    4712 generic.go:356] "Generic (PLEG): container finished" podID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerID="1365a70761faa306c574957470df665ddeafe740d8d77e01d5f213c474d5f470" exitCode=0
Oct 30 19:31:08.303177 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.303154    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd" event={"ID":"d6ddfa89-9f31-4674-a41e-bcc43761b954","Type":"ContainerDied","Data":"1365a70761faa306c574957470df665ddeafe740d8d77e01d5f213c474d5f470"}
Oct 30 19:31:08.303734 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.303712    4712 scope.go:122] "RemoveContainer" containerID="1365a70761faa306c574957470df665ddeafe740d8d77e01d5f213c474d5f470"
Oct 30 19:31:08.306338 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.306295    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:31:08.306464 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.306414    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:31:08.306535 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.306467    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:31:08.656853 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.656805    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:08.803790 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.803741    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:08.968929 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:08.966871    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:09.353560 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:09.352476    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc" event={"ID":"b727df7a-56c7-46fc-974a-38aebc2593c3","Type":"ContainerStarted","Data":"b13b462d93ac4b7cea8d62b6520164d674c301d00fc903e9b6e063ed7b4e5d03"}
Oct 30 19:31:09.368472 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:09.356781    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd" event={"ID":"d6ddfa89-9f31-4674-a41e-bcc43761b954","Type":"ContainerStarted","Data":"b9527213d72efc1639e36c1981b3bfb56dd0473067d7813df820fd7a6242fd64"}
Oct 30 19:31:09.625449 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:09.623823    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:31:09.625449 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:09.623939    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 19:31:09.625449 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:09.623977    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd" podUID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 19:31:09.984243 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:09.984147    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:10.187094 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:10.187058    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:10.299493 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:10.299087    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:31:10.299493 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:10.299179    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:31:10.299493 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:10.299210    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc" podUID="b727df7a-56c7-46fc-974a-38aebc2593c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 19:31:10.629177 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:10.628731    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 19:31:10.629177 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:10.628792    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd" podUID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 19:31:11.283673 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:11.283629    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"
Oct 30 19:31:11.299411 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:11.299352    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:31:11.299565 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:11.299432    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc" podUID="b727df7a-56c7-46fc-974a-38aebc2593c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 19:31:11.629382 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:11.624774    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 19:31:11.629382 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:11.624838    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd" podUID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 19:31:12.251793 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:12.251728    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:12.300048 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:12.300016    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:31:12.300241 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:12.300086    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc" podUID="b727df7a-56c7-46fc-974a-38aebc2593c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 19:31:12.631384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:12.624682    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 19:31:12.631384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:12.624741    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd" podUID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 19:31:12.753473 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:12.751520    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:13.225385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:13.225278    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:13.299479 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:13.299417    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:31:13.299644 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:13.299505    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc" podUID="b727df7a-56c7-46fc-974a-38aebc2593c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 19:31:13.386841 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:13.386808    4712 generic.go:356] "Generic (PLEG): container finished" podID="045af806-47d7-41b6-9cd7-ab0b9c851f3c" containerID="d4d13207653fbc31f850ff0f17e0e35840f4488515bdad29a82d1b7357365a07" exitCode=0
Oct 30 19:31:13.386998 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:13.386880    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst" event={"ID":"045af806-47d7-41b6-9cd7-ab0b9c851f3c","Type":"ContainerDied","Data":"d4d13207653fbc31f850ff0f17e0e35840f4488515bdad29a82d1b7357365a07"}
Oct 30 19:31:13.387259 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:13.387242    4712 scope.go:122] "RemoveContainer" containerID="d4d13207653fbc31f850ff0f17e0e35840f4488515bdad29a82d1b7357365a07"
Oct 30 19:31:13.388666 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:13.388633    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:31:13.388819 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:13.388805    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:31:13.388910 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:13.388898    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:31:13.636217 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:13.624643    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 19:31:13.636217 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:13.624699    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd" podUID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 19:31:13.915867 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:13.915836    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:14.071613 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:14.071576    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:14.267134 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:14.267034    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:14.300152 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:14.300116    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:31:14.300331 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:14.300171    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc" podUID="b727df7a-56c7-46fc-974a-38aebc2593c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 19:31:14.395426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:14.395395    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst" event={"ID":"045af806-47d7-41b6-9cd7-ab0b9c851f3c","Type":"ContainerStarted","Data":"e469737daf6cc305c44bf5292b0d9161b906490879a98081f4df75c7ad4067fa"}
Oct 30 19:31:14.397740 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:14.397700    4712 generic.go:356] "Generic (PLEG): container finished" podID="3fce9cac-38e0-436f-a4aa-4f82b908e11e" containerID="a2d80302043fb98ff8f87b011a770cfa15237d7ea139100c7a7ef665859d1421" exitCode=0
Oct 30 19:31:14.398014 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:14.397998    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp" event={"ID":"3fce9cac-38e0-436f-a4aa-4f82b908e11e","Type":"ContainerDied","Data":"a2d80302043fb98ff8f87b011a770cfa15237d7ea139100c7a7ef665859d1421"}
Oct 30 19:31:14.398559 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:14.398542    4712 scope.go:122] "RemoveContainer" containerID="a2d80302043fb98ff8f87b011a770cfa15237d7ea139100c7a7ef665859d1421"
Oct 30 19:31:14.400231 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:14.400198    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:31:14.400321 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:14.400267    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:31:14.400321 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:14.400299    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:31:14.434931 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:14.434903    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:31:14.435282 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:14.435258    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:31:14.435461 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:14.435432    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst" podUID="045af806-47d7-41b6-9cd7-ab0b9c851f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 19:31:14.624925 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:14.624884    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 19:31:14.625188 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:14.625155    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd" podUID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 19:31:14.900950 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:14.900909    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:15.075176 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:15.075139    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:15.303674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:15.303604    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:31:15.303900 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:15.303874    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:31:15.309176 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:15.309149    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:31:15.418383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:15.418331    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:31:15.421246 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:15.420544    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp" event={"ID":"3fce9cac-38e0-436f-a4aa-4f82b908e11e","Type":"ContainerStarted","Data":"1fd0a1b7ca423ef3634ec9d8ae6a6f7ee03cdaf34f63a430faee2f5bc16da024"}
Oct 30 19:31:15.422136 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:15.422111    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst" podUID="045af806-47d7-41b6-9cd7-ab0b9c851f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 19:31:15.659387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:15.640623    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:31:15.659870 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:15.659843    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:31:15.661452 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:15.661418    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:31:16.182475 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:16.182435    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:31:16.183423 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:16.183348    4712 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 19:31:16.183513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:16.183461    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp" podUID="3fce9cac-38e0-436f-a4aa-4f82b908e11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 19:31:16.252772 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:16.252734    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:16.424889 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:16.422749    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:31:16.424889 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:16.422827    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst" podUID="045af806-47d7-41b6-9cd7-ab0b9c851f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 19:31:17.126473 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:17.126443    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:31:17.127452 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:17.127431    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:31:17.182215 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:17.182188    4712 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 19:31:17.182469 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:17.182443    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp" podUID="3fce9cac-38e0-436f-a4aa-4f82b908e11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 19:31:17.427454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:17.423734    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:31:17.427454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:17.423792    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst" podUID="045af806-47d7-41b6-9cd7-ab0b9c851f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 19:31:18.029562 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:18.029512    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:18.182020 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:18.181991    4712 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 19:31:18.182239 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:18.182218    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp" podUID="3fce9cac-38e0-436f-a4aa-4f82b908e11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 19:31:18.457865 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:18.457824    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:31:18.458461 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:18.458429    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst" podUID="045af806-47d7-41b6-9cd7-ab0b9c851f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 19:31:19.182730 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:19.182695    4712 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 19:31:19.182899 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:19.182749    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp" podUID="3fce9cac-38e0-436f-a4aa-4f82b908e11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 19:31:19.255385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:19.251758    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:19.417707 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:19.417670    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:31:19.417993 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:19.417966    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst" podUID="045af806-47d7-41b6-9cd7-ab0b9c851f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 19:31:19.838057 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:19.838015    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:20.055581 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:20.055549    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:20.182631 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:20.182600    4712 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 19:31:20.182805 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:20.182652    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp" podUID="3fce9cac-38e0-436f-a4aa-4f82b908e11e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 19:31:20.262614 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:20.262583    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:20.433974 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:20.433895    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:31:20.435225 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:20.435201    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:31:21.186032 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:21.186001    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:31:21.187086 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:21.187067    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:31:21.203681 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:21.203657    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:22.069792 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:22.069746    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:23.395713 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:23.395677    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:23.922664 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:23.922626    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:24.119493 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:24.119461    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:24.303230 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:24.303131    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:24.463051 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:24.463012    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:25.061012 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:25.060977    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:25.256058 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:25.256023    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:25.893540 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:25.893489    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:26.124246 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:26.124209    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:27.443531 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:27.443494    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:27.892920 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:27.892887    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"]
Oct 30 19:31:27.899550 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:27.897730    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:27.931749 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:27.931725    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3735b2\"/\"openshift-service-ca.crt\""
Oct 30 19:31:27.931749 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:27.931743    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3735b2\"/\"kube-root-ca.crt\""
Oct 30 19:31:27.931968 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:27.931802    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3735b2\"/\"default-dockercfg-w7mcc\""
Oct 30 19:31:27.931968 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:27.931815    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3735b2\"/\"simulator-w2-3735b2-kserve-self-signed-certs\""
Oct 30 19:31:27.935069 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:27.935047    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"]
Oct 30 19:31:27.994540 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:27.994513    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-home\") pod \"simulator-w2-3735b2-kserve-54699dfd4c-mmklj\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") " pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:27.994677 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:27.994545    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-model-cache\") pod \"simulator-w2-3735b2-kserve-54699dfd4c-mmklj\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") " pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:27.994677 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:27.994570    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-dshm\") pod \"simulator-w2-3735b2-kserve-54699dfd4c-mmklj\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") " pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:27.994677 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:27.994642    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12d39453-861c-437a-8bbe-e9942b8778d3-tls-certs\") pod \"simulator-w2-3735b2-kserve-54699dfd4c-mmklj\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") " pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:27.994814 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:27.994719    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srw2b\" (UniqueName: \"kubernetes.io/projected/12d39453-861c-437a-8bbe-e9942b8778d3-kube-api-access-srw2b\") pod \"simulator-w2-3735b2-kserve-54699dfd4c-mmklj\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") " pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:27.994856 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:27.994829    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-tmp-dir\") pod \"simulator-w2-3735b2-kserve-54699dfd4c-mmklj\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") " pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:28.011971 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:28.011936    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:28.095892 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:28.095867    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-tmp-dir\") pod \"simulator-w2-3735b2-kserve-54699dfd4c-mmklj\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") " pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:28.095999 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:28.095909    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-home\") pod \"simulator-w2-3735b2-kserve-54699dfd4c-mmklj\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") " pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:28.095999 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:28.095929    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-model-cache\") pod \"simulator-w2-3735b2-kserve-54699dfd4c-mmklj\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") " pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:28.095999 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:28.095959    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-dshm\") pod \"simulator-w2-3735b2-kserve-54699dfd4c-mmklj\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") " pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:28.095999 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:28.095981    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12d39453-861c-437a-8bbe-e9942b8778d3-tls-certs\") pod \"simulator-w2-3735b2-kserve-54699dfd4c-mmklj\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") " pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:28.096172 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:28.096022    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srw2b\" (UniqueName: \"kubernetes.io/projected/12d39453-861c-437a-8bbe-e9942b8778d3-kube-api-access-srw2b\") pod \"simulator-w2-3735b2-kserve-54699dfd4c-mmklj\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") " pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:28.096343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:28.096320    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-tmp-dir\") pod \"simulator-w2-3735b2-kserve-54699dfd4c-mmklj\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") " pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:28.096455 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:28.096385    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-home\") pod \"simulator-w2-3735b2-kserve-54699dfd4c-mmklj\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") " pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:28.096455 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:28.096394    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-model-cache\") pod \"simulator-w2-3735b2-kserve-54699dfd4c-mmklj\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") " pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:28.098395 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:28.098375    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-dshm\") pod \"simulator-w2-3735b2-kserve-54699dfd4c-mmklj\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") " pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:28.098532 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:28.098518    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12d39453-861c-437a-8bbe-e9942b8778d3-tls-certs\") pod \"simulator-w2-3735b2-kserve-54699dfd4c-mmklj\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") " pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:28.114730 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:28.114711    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srw2b\" (UniqueName: \"kubernetes.io/projected/12d39453-861c-437a-8bbe-e9942b8778d3-kube-api-access-srw2b\") pod \"simulator-w2-3735b2-kserve-54699dfd4c-mmklj\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") " pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:28.213251 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:28.213173    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:28.505001 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:28.504976    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"]
Oct 30 19:31:28.506212 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:31:28.506187    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12d39453_861c_437a_8bbe_e9942b8778d3.slice/crio-a8b8e952cbed78e359f340935c796bcf29e4f3931400296e79019146961f54f5 WatchSource:0}: Error finding container a8b8e952cbed78e359f340935c796bcf29e4f3931400296e79019146961f54f5: Status 404 returned error can't find the container with id a8b8e952cbed78e359f340935c796bcf29e4f3931400296e79019146961f54f5
Oct 30 19:31:28.522634 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:28.522604    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" event={"ID":"12d39453-861c-437a-8bbe-e9942b8778d3","Type":"ContainerStarted","Data":"a8b8e952cbed78e359f340935c796bcf29e4f3931400296e79019146961f54f5"}
Oct 30 19:31:28.914336 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:28.914302    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:29.529686 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:29.529653    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" event={"ID":"12d39453-861c-437a-8bbe-e9942b8778d3","Type":"ContainerStarted","Data":"6a99b12677b8af0c56071f1f82b636ef5cdf3d052f0674136ce8ac3cae7dca0f"}
Oct 30 19:31:29.571402 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:29.571337    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podStartSLOduration=2.571321772 podStartE2EDuration="2.571321772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:29.570490743 +0000 UTC m=+642.869752884" watchObservedRunningTime="2026-10-30 19:31:29.571321772 +0000 UTC m=+642.870583901"
Oct 30 19:31:30.213289 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:30.213258    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:30.473195 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:30.473092    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:30.689699 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:30.689660    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:31.877540 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:31.877506    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"]
Oct 30 19:31:31.880738 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:31.880713    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:31.898289 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:31.898263    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4b4c34\"/\"kube-root-ca.crt\""
Oct 30 19:31:31.900611 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:31.900577    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:31.910175 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:31.910153    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4b4c34\"/\"openshift-service-ca.crt\""
Oct 30 19:31:31.910275 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:31.910242    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b4c34\"/\"default-dockercfg-499bb\""
Oct 30 19:31:31.910454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:31.910436    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b4c34\"/\"simulator-w0-4b4c34-kserve-self-signed-certs\""
Oct 30 19:31:31.912593 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:31.912574    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"]
Oct 30 19:31:32.035537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.035509    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-dshm\") pod \"simulator-w0-4b4c34-kserve-f5fcbb459-zcclk\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") " pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.035706 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.035546    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-model-cache\") pod \"simulator-w0-4b4c34-kserve-f5fcbb459-zcclk\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") " pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.035706 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.035591    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a3b2eb-8722-4782-af07-70239b3c0ff5-tls-certs\") pod \"simulator-w0-4b4c34-kserve-f5fcbb459-zcclk\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") " pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.035706 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.035677    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-tmp-dir\") pod \"simulator-w0-4b4c34-kserve-f5fcbb459-zcclk\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") " pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.035854 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.035759    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l499v\" (UniqueName: \"kubernetes.io/projected/d1a3b2eb-8722-4782-af07-70239b3c0ff5-kube-api-access-l499v\") pod \"simulator-w0-4b4c34-kserve-f5fcbb459-zcclk\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") " pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.035854 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.035825    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-home\") pod \"simulator-w0-4b4c34-kserve-f5fcbb459-zcclk\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") " pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.046478 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.046454    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:32.136836 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.136750    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-home\") pod \"simulator-w0-4b4c34-kserve-f5fcbb459-zcclk\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") " pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.136836 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.136813    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-dshm\") pod \"simulator-w0-4b4c34-kserve-f5fcbb459-zcclk\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") " pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.137236 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.136841    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-model-cache\") pod \"simulator-w0-4b4c34-kserve-f5fcbb459-zcclk\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") " pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.137236 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.136923    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a3b2eb-8722-4782-af07-70239b3c0ff5-tls-certs\") pod \"simulator-w0-4b4c34-kserve-f5fcbb459-zcclk\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") " pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.137236 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.136956    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-tmp-dir\") pod \"simulator-w0-4b4c34-kserve-f5fcbb459-zcclk\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") " pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.137236 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.137013    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l499v\" (UniqueName: \"kubernetes.io/projected/d1a3b2eb-8722-4782-af07-70239b3c0ff5-kube-api-access-l499v\") pod \"simulator-w0-4b4c34-kserve-f5fcbb459-zcclk\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") " pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.144029 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.137821    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-model-cache\") pod \"simulator-w0-4b4c34-kserve-f5fcbb459-zcclk\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") " pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.144029 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.137925    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-home\") pod \"simulator-w0-4b4c34-kserve-f5fcbb459-zcclk\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") " pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.144029 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.138161    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-tmp-dir\") pod \"simulator-w0-4b4c34-kserve-f5fcbb459-zcclk\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") " pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.144029 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.140516    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-dshm\") pod \"simulator-w0-4b4c34-kserve-f5fcbb459-zcclk\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") " pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.146008 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.145985    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a3b2eb-8722-4782-af07-70239b3c0ff5-tls-certs\") pod \"simulator-w0-4b4c34-kserve-f5fcbb459-zcclk\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") " pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.157123 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.157103    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l499v\" (UniqueName: \"kubernetes.io/projected/d1a3b2eb-8722-4782-af07-70239b3c0ff5-kube-api-access-l499v\") pod \"simulator-w0-4b4c34-kserve-f5fcbb459-zcclk\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") " pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.193347 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.193323    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:32.486423 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.486399    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"]
Oct 30 19:31:32.486785 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:31:32.486762    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1a3b2eb_8722_4782_af07_70239b3c0ff5.slice/crio-33b4e263125708550639faf29bc78d4d63860459b5f69f4388906bf51db91293 WatchSource:0}: Error finding container 33b4e263125708550639faf29bc78d4d63860459b5f69f4388906bf51db91293: Status 404 returned error can't find the container with id 33b4e263125708550639faf29bc78d4d63860459b5f69f4388906bf51db91293
Oct 30 19:31:32.552675 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.552645    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" event={"ID":"d1a3b2eb-8722-4782-af07-70239b3c0ff5","Type":"ContainerStarted","Data":"b715774721260001c1debf173fc547a728f359b9dc8ac5b2465eb7ffeb926069"}
Oct 30 19:31:32.552769 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.552693    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" event={"ID":"d1a3b2eb-8722-4782-af07-70239b3c0ff5","Type":"ContainerStarted","Data":"33b4e263125708550639faf29bc78d4d63860459b5f69f4388906bf51db91293"}
Oct 30 19:31:32.596384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:32.596326    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podStartSLOduration=1.596311571 podStartE2EDuration="1.596311571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:32.594043425 +0000 UTC m=+645.893305554" watchObservedRunningTime="2026-10-30 19:31:32.596311571 +0000 UTC m=+645.895573700"
Oct 30 19:31:33.194106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:33.194076    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:33.380136 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:33.379895    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:34.781876 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:34.781834    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:36.286854 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:36.286806    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:36.372866 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:31:36.372830    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:31:36.434493 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:36.434462    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:37.893012 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:37.892892    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:38.213680 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:38.213599    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:38.225799 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:38.225771    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:38.591996 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:38.591911    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:31:39.376972 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:39.376933    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:40.898692 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:40.898643    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:42.193965 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.193934    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:42.206572 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.206548    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:42.328515 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.328485    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"]
Oct 30 19:31:42.331752 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.331730    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.346431 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.346406    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-tmp-dir\") pod \"simulator-w3-dd23d1-kserve-79c66fc974-rrpnr\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") " pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.346569 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.346443    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0615cd7-08e0-4930-937c-8027e74e57e2-tls-certs\") pod \"simulator-w3-dd23d1-kserve-79c66fc974-rrpnr\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") " pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.346569 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.346538    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-dshm\") pod \"simulator-w3-dd23d1-kserve-79c66fc974-rrpnr\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") " pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.346569 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.346566    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-model-cache\") pod \"simulator-w3-dd23d1-kserve-79c66fc974-rrpnr\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") " pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.346734 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.346609    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-home\") pod \"simulator-w3-dd23d1-kserve-79c66fc974-rrpnr\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") " pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.346795 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.346735    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4qpl\" (UniqueName: \"kubernetes.io/projected/b0615cd7-08e0-4930-937c-8027e74e57e2-kube-api-access-k4qpl\") pod \"simulator-w3-dd23d1-kserve-79c66fc974-rrpnr\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") " pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.350163 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.350141    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-dd23d1\"/\"openshift-service-ca.crt\""
Oct 30 19:31:42.350300 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.350275    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd23d1\"/\"default-dockercfg-tc9vt\""
Oct 30 19:31:42.350498 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.350482    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-dd23d1\"/\"kube-root-ca.crt\""
Oct 30 19:31:42.350597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.350577    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd23d1\"/\"simulator-w3-dd23d1-kserve-self-signed-certs\""
Oct 30 19:31:42.353098 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.353077    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"]
Oct 30 19:31:42.443395 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.443347    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:42.447648 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.447588    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-tmp-dir\") pod \"simulator-w3-dd23d1-kserve-79c66fc974-rrpnr\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") " pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.447648 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.447630    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0615cd7-08e0-4930-937c-8027e74e57e2-tls-certs\") pod \"simulator-w3-dd23d1-kserve-79c66fc974-rrpnr\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") " pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.447829 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.447669    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-dshm\") pod \"simulator-w3-dd23d1-kserve-79c66fc974-rrpnr\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") " pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.447890 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.447844    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-model-cache\") pod \"simulator-w3-dd23d1-kserve-79c66fc974-rrpnr\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") " pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.447953 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.447939    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-home\") pod \"simulator-w3-dd23d1-kserve-79c66fc974-rrpnr\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") " pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.448040 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.448015    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4qpl\" (UniqueName: \"kubernetes.io/projected/b0615cd7-08e0-4930-937c-8027e74e57e2-kube-api-access-k4qpl\") pod \"simulator-w3-dd23d1-kserve-79c66fc974-rrpnr\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") " pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.448118 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.448037    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-tmp-dir\") pod \"simulator-w3-dd23d1-kserve-79c66fc974-rrpnr\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") " pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.448229 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.448208    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-model-cache\") pod \"simulator-w3-dd23d1-kserve-79c66fc974-rrpnr\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") " pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.448389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.448345    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-home\") pod \"simulator-w3-dd23d1-kserve-79c66fc974-rrpnr\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") " pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.450012 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.449990    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-dshm\") pod \"simulator-w3-dd23d1-kserve-79c66fc974-rrpnr\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") " pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.450272 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.450256    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0615cd7-08e0-4930-937c-8027e74e57e2-tls-certs\") pod \"simulator-w3-dd23d1-kserve-79c66fc974-rrpnr\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") " pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.467455 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.467432    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4qpl\" (UniqueName: \"kubernetes.io/projected/b0615cd7-08e0-4930-937c-8027e74e57e2-kube-api-access-k4qpl\") pod \"simulator-w3-dd23d1-kserve-79c66fc974-rrpnr\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") " pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.589037 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.588998    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:42.613514 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.613483    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:31:42.646331 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.646303    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:42.945017 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:42.944994    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"]
Oct 30 19:31:42.946182 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:31:42.946156    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0615cd7_08e0_4930_937c_8027e74e57e2.slice/crio-2f4517efe3ec1f73cfe93f5e7d1510642da03cec3c3e5bc909bbade171e4c8cb WatchSource:0}: Error finding container 2f4517efe3ec1f73cfe93f5e7d1510642da03cec3c3e5bc909bbade171e4c8cb: Status 404 returned error can't find the container with id 2f4517efe3ec1f73cfe93f5e7d1510642da03cec3c3e5bc909bbade171e4c8cb
Oct 30 19:31:43.609004 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:43.608968    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" event={"ID":"b0615cd7-08e0-4930-937c-8027e74e57e2","Type":"ContainerStarted","Data":"9684c490762d9b811eb286d3a81102c7278a859d8edadd2753caa612176dddef"}
Oct 30 19:31:43.609423 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:43.609013    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" event={"ID":"b0615cd7-08e0-4930-937c-8027e74e57e2","Type":"ContainerStarted","Data":"2f4517efe3ec1f73cfe93f5e7d1510642da03cec3c3e5bc909bbade171e4c8cb"}
Oct 30 19:31:43.646725 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:43.646701    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:43.650106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:43.650067    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podStartSLOduration=1.650056025 podStartE2EDuration="1.650056025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:43.649436298 +0000 UTC m=+656.948698424" watchObservedRunningTime="2026-10-30 19:31:43.650056025 +0000 UTC m=+656.949318154"
Oct 30 19:31:43.991760 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:43.991711    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:44.213502 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:44.213465    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:45.460426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:45.460391    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:46.888697 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:46.888661    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:47.361962 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:47.361888    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:48.466206 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:48.466160    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:49.862348 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:49.862314    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:51.263584 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:51.263544    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:52.646916 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:52.646883    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:52.658934 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:52.658911    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:52.669887 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:52.669863    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:31:52.798607 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:52.798573    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:54.350321 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:54.350288    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:55.872891 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:55.872854    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:56.829140 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:56.829109    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:57.381309 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:57.381264    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:31:58.978641 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:31:58.978546    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:00.465944 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:00.465900    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:02.068747 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:02.068713    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:03.466374 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:03.466313    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:04.885907 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:04.885877    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:06.373079 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:32:06.373044    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:32:06.388859 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:06.386113    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:07.131161 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.131124    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"]
Oct 30 19:32:07.134490 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.134465    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.157062 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.157035    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"]
Oct 30 19:32:07.165667 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.165645    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-544fc8\"/\"kube-root-ca.crt\""
Oct 30 19:32:07.165964 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.165950    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-544fc8\"/\"default-dockercfg-g5jtg\""
Oct 30 19:32:07.166134 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.166121    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-544fc8\"/\"openshift-service-ca.crt\""
Oct 30 19:32:07.166253 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.166230    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-544fc8\"/\"premium-w5-544fc8-kserve-self-signed-certs\""
Oct 30 19:32:07.174828 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.174790    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h95hb\" (UniqueName: \"kubernetes.io/projected/0a6dccf3-9a50-43ef-833f-1e66fe53890f-kube-api-access-h95hb\") pod \"premium-w5-544fc8-kserve-554c74c5d9-2gmk8\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") " pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.174927 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.174836    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-home\") pod \"premium-w5-544fc8-kserve-554c74c5d9-2gmk8\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") " pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.174927 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.174854    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-dshm\") pod \"premium-w5-544fc8-kserve-554c74c5d9-2gmk8\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") " pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.175049 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.174985    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-tmp-dir\") pod \"premium-w5-544fc8-kserve-554c74c5d9-2gmk8\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") " pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.175102 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.175070    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-model-cache\") pod \"premium-w5-544fc8-kserve-554c74c5d9-2gmk8\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") " pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.175152 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.175139    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a6dccf3-9a50-43ef-833f-1e66fe53890f-tls-certs\") pod \"premium-w5-544fc8-kserve-554c74c5d9-2gmk8\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") " pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.236824 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.236780    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:07.275772 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.275746    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h95hb\" (UniqueName: \"kubernetes.io/projected/0a6dccf3-9a50-43ef-833f-1e66fe53890f-kube-api-access-h95hb\") pod \"premium-w5-544fc8-kserve-554c74c5d9-2gmk8\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") " pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.275903 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.275781    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-home\") pod \"premium-w5-544fc8-kserve-554c74c5d9-2gmk8\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") " pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.275903 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.275806    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-dshm\") pod \"premium-w5-544fc8-kserve-554c74c5d9-2gmk8\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") " pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.275903 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.275840    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-tmp-dir\") pod \"premium-w5-544fc8-kserve-554c74c5d9-2gmk8\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") " pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.276093 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.275906    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-model-cache\") pod \"premium-w5-544fc8-kserve-554c74c5d9-2gmk8\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") " pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.276093 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.275958    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a6dccf3-9a50-43ef-833f-1e66fe53890f-tls-certs\") pod \"premium-w5-544fc8-kserve-554c74c5d9-2gmk8\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") " pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.276196 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.276170    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-home\") pod \"premium-w5-544fc8-kserve-554c74c5d9-2gmk8\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") " pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.276327 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.276304    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-tmp-dir\") pod \"premium-w5-544fc8-kserve-554c74c5d9-2gmk8\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") " pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.276424 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.276377    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-model-cache\") pod \"premium-w5-544fc8-kserve-554c74c5d9-2gmk8\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") " pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.278037 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.278015    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-dshm\") pod \"premium-w5-544fc8-kserve-554c74c5d9-2gmk8\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") " pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.278353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.278335    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a6dccf3-9a50-43ef-833f-1e66fe53890f-tls-certs\") pod \"premium-w5-544fc8-kserve-554c74c5d9-2gmk8\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") " pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.294930 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.294903    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h95hb\" (UniqueName: \"kubernetes.io/projected/0a6dccf3-9a50-43ef-833f-1e66fe53890f-kube-api-access-h95hb\") pod \"premium-w5-544fc8-kserve-554c74c5d9-2gmk8\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") " pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.459874 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.459793    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:07.753314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.753286    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"]
Oct 30 19:32:07.755804 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:32:07.755777    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a6dccf3_9a50_43ef_833f_1e66fe53890f.slice/crio-d660850c639aace98a2307bfe9973782dc3066a21edd2ae441da2550d37a19cd WatchSource:0}: Error finding container d660850c639aace98a2307bfe9973782dc3066a21edd2ae441da2550d37a19cd: Status 404 returned error can't find the container with id d660850c639aace98a2307bfe9973782dc3066a21edd2ae441da2550d37a19cd
Oct 30 19:32:07.977599 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:07.977547    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:08.746654 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:08.746614    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" event={"ID":"0a6dccf3-9a50-43ef-833f-1e66fe53890f","Type":"ContainerStarted","Data":"3713b43ac06964f79842389acdb50f7b1938cc86ac407a27212bcaac43fbceeb"}
Oct 30 19:32:08.747030 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:08.746669    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" event={"ID":"0a6dccf3-9a50-43ef-833f-1e66fe53890f","Type":"ContainerStarted","Data":"d660850c639aace98a2307bfe9973782dc3066a21edd2ae441da2550d37a19cd"}
Oct 30 19:32:08.785255 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:08.785200    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podStartSLOduration=1.785187903 podStartE2EDuration="1.785187903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:08.784262679 +0000 UTC m=+682.083524809" watchObservedRunningTime="2026-10-30 19:32:08.785187903 +0000 UTC m=+682.084450032"
Oct 30 19:32:08.902420 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:08.902388    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:09.460604 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:09.460562    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:09.559342 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:09.559312    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:10.999411 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:10.999353    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:12.062106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:12.062075    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:12.460284 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:12.460245    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:13.894193 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:13.894113    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:15.467418 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:15.467326    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:17.093550 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:17.093492    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:17.460749 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:17.460720    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:17.473168 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:17.473146    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:17.806053 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:17.805979    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:32:18.296378 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.296313    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"]
Oct 30 19:32:18.299509 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.299487    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.327207 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.327185    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d797d2\"/\"kube-root-ca.crt\""
Oct 30 19:32:18.327517 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.327499    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d797d2\"/\"premium-w1-d797d2-kserve-self-signed-certs\""
Oct 30 19:32:18.330781 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.330759    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d797d2\"/\"default-dockercfg-mlm87\""
Oct 30 19:32:18.330781 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.330772    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d797d2\"/\"openshift-service-ca.crt\""
Oct 30 19:32:18.332938 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.332917    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"]
Oct 30 19:32:18.400881 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.400852    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3bee1cf-6278-4666-93db-a483e7dca394-tls-certs\") pod \"premium-w1-d797d2-kserve-78c7b9b848-bmvv8\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") " pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.400995 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.400901    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-tmp-dir\") pod \"premium-w1-d797d2-kserve-78c7b9b848-bmvv8\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") " pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.400995 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.400977    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-home\") pod \"premium-w1-d797d2-kserve-78c7b9b848-bmvv8\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") " pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.401108 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.401039    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpcxh\" (UniqueName: \"kubernetes.io/projected/d3bee1cf-6278-4666-93db-a483e7dca394-kube-api-access-tpcxh\") pod \"premium-w1-d797d2-kserve-78c7b9b848-bmvv8\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") " pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.401226 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.401206    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-dshm\") pod \"premium-w1-d797d2-kserve-78c7b9b848-bmvv8\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") " pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.401287 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.401244    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-model-cache\") pod \"premium-w1-d797d2-kserve-78c7b9b848-bmvv8\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") " pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.411865 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.411838    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:18.507778 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.507744    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-dshm\") pod \"premium-w1-d797d2-kserve-78c7b9b848-bmvv8\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") " pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.508035 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.508014    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-model-cache\") pod \"premium-w1-d797d2-kserve-78c7b9b848-bmvv8\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") " pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.508218 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.508174    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3bee1cf-6278-4666-93db-a483e7dca394-tls-certs\") pod \"premium-w1-d797d2-kserve-78c7b9b848-bmvv8\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") " pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.508353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.508337    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-tmp-dir\") pod \"premium-w1-d797d2-kserve-78c7b9b848-bmvv8\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") " pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.508500 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.508484    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-home\") pod \"premium-w1-d797d2-kserve-78c7b9b848-bmvv8\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") " pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.508687 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.508670    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpcxh\" (UniqueName: \"kubernetes.io/projected/d3bee1cf-6278-4666-93db-a483e7dca394-kube-api-access-tpcxh\") pod \"premium-w1-d797d2-kserve-78c7b9b848-bmvv8\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") " pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.508853 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.508740    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-tmp-dir\") pod \"premium-w1-d797d2-kserve-78c7b9b848-bmvv8\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") " pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.508957 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.508505    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-model-cache\") pod \"premium-w1-d797d2-kserve-78c7b9b848-bmvv8\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") " pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.509096 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.509070    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-home\") pod \"premium-w1-d797d2-kserve-78c7b9b848-bmvv8\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") " pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.515261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.515237    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-dshm\") pod \"premium-w1-d797d2-kserve-78c7b9b848-bmvv8\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") " pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.515732 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.515616    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3bee1cf-6278-4666-93db-a483e7dca394-tls-certs\") pod \"premium-w1-d797d2-kserve-78c7b9b848-bmvv8\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") " pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.528490 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.528465    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpcxh\" (UniqueName: \"kubernetes.io/projected/d3bee1cf-6278-4666-93db-a483e7dca394-kube-api-access-tpcxh\") pod \"premium-w1-d797d2-kserve-78c7b9b848-bmvv8\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") " pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.612057 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.611985    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:18.684577 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.684542    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:18.909913 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:18.909884    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"]
Oct 30 19:32:18.913670 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:32:18.913638    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3bee1cf_6278_4666_93db_a483e7dca394.slice/crio-aafe1208dafa3144ef389ca71a291992b2145c2132f3d3943b8318fc0a3c109d WatchSource:0}: Error finding container aafe1208dafa3144ef389ca71a291992b2145c2132f3d3943b8318fc0a3c109d: Status 404 returned error can't find the container with id aafe1208dafa3144ef389ca71a291992b2145c2132f3d3943b8318fc0a3c109d
Oct 30 19:32:19.808052 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:19.808015    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" event={"ID":"d3bee1cf-6278-4666-93db-a483e7dca394","Type":"ContainerStarted","Data":"b57bb798904714d0e87194dc71c0aa595fcc5dc1f7963bfb16fef73d8614f26d"}
Oct 30 19:32:19.808556 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:19.808065    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" event={"ID":"d3bee1cf-6278-4666-93db-a483e7dca394","Type":"ContainerStarted","Data":"aafe1208dafa3144ef389ca71a291992b2145c2132f3d3943b8318fc0a3c109d"}
Oct 30 19:32:19.851499 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:19.851456    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podStartSLOduration=1.8514452380000002 podStartE2EDuration="1.851445238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:19.85082784 +0000 UTC m=+693.150089969" watchObservedRunningTime="2026-10-30 19:32:19.851445238 +0000 UTC m=+693.150707367"
Oct 30 19:32:20.612613 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:20.612580    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:21.425226 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:21.425195    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:21.595503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:21.595474    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:22.092510 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:22.092470    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:23.320640 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:23.320598    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:23.694036 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:23.693996    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:25.088626 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:25.088580    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:26.577777 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:26.577730    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:28.035495 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:28.035458    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:28.612776 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:28.612743    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:28.626000 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:28.625971    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:28.873255 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:28.873227    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:32:29.802113 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:29.802073    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:30.847209 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:30.847176    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:31.505031 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:31.504991    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:33.199794 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:33.199760    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:34.684794 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:34.684751    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:36.081060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:36.081022    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:36.372509 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:32:36.372473    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:32:37.671530 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:37.671480    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:39.242358 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:39.242321    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:40.996138 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:40.996101    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:42.520333 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:42.520249    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:44.180412 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:44.180377    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:45.904123 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:45.904075    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:47.620634 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:47.620597    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:49.120256 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:49.120224    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:50.921280 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:50.921242    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:52.483979 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:52.483916    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:54.098620 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:54.098576    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:55.682687 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:55.682655    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:57.280076 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:57.280029    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:32:58.918946 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:32:58.918902    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:00.388034 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:00.387998    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:02.195569 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:02.195528    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:03.772779 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:03.772711    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:05.437657 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:05.437594    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:06.372807 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:33:06.372771    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:33:07.119370 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:07.119332    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:08.798429 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:08.798356    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:10.416486 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:10.416453    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:12.097329 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:12.097288    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:13.578160 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:13.578121    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:15.085680 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:15.085614    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:16.820750 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:16.820714    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:18.622823 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:18.622778    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:20.378736 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:20.378704    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:22.117542 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:22.117503    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:23.949060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:23.949029    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:25.613212 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:25.613177    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:27.394338 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:27.394304    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:28.889401 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:28.889297    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:30.391087 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:30.391044    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:30.936566 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:30.936534    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:32.782810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:32.782765    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:34.227078 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.227045    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"]
Oct 30 19:33:34.301309 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.301278    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"]
Oct 30 19:33:34.301496 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.301413    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.324703 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.324677    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b4c34\"/\"distinct-w0-4b4c34-kserve-self-signed-certs\""
Oct 30 19:33:34.386545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.386342    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:34.395412 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.395321    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-dshm\") pod \"distinct-w0-4b4c34-kserve-66c49c5849-55dlq\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.395412 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.395373    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-tls-certs\") pod \"distinct-w0-4b4c34-kserve-66c49c5849-55dlq\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.395412 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.395394    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-home\") pod \"distinct-w0-4b4c34-kserve-66c49c5849-55dlq\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.395597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.395499    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-tmp-dir\") pod \"distinct-w0-4b4c34-kserve-66c49c5849-55dlq\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.395597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.395586    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-model-cache\") pod \"distinct-w0-4b4c34-kserve-66c49c5849-55dlq\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.395681 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.395667    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdqm8\" (UniqueName: \"kubernetes.io/projected/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-kube-api-access-jdqm8\") pod \"distinct-w0-4b4c34-kserve-66c49c5849-55dlq\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.497597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.496779    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdqm8\" (UniqueName: \"kubernetes.io/projected/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-kube-api-access-jdqm8\") pod \"distinct-w0-4b4c34-kserve-66c49c5849-55dlq\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.497597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.496827    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-dshm\") pod \"distinct-w0-4b4c34-kserve-66c49c5849-55dlq\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.497597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.496857    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-tls-certs\") pod \"distinct-w0-4b4c34-kserve-66c49c5849-55dlq\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.497597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.496873    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-home\") pod \"distinct-w0-4b4c34-kserve-66c49c5849-55dlq\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.497597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.496895    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-tmp-dir\") pod \"distinct-w0-4b4c34-kserve-66c49c5849-55dlq\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.497597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.496934    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-model-cache\") pod \"distinct-w0-4b4c34-kserve-66c49c5849-55dlq\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.497597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.497303    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-model-cache\") pod \"distinct-w0-4b4c34-kserve-66c49c5849-55dlq\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.498118 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.497986    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-home\") pod \"distinct-w0-4b4c34-kserve-66c49c5849-55dlq\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.498751 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.498723    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-tmp-dir\") pod \"distinct-w0-4b4c34-kserve-66c49c5849-55dlq\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.501418 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.501345    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-tls-certs\") pod \"distinct-w0-4b4c34-kserve-66c49c5849-55dlq\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.503572 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.503552    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-dshm\") pod \"distinct-w0-4b4c34-kserve-66c49c5849-55dlq\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.548381 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.548049    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdqm8\" (UniqueName: \"kubernetes.io/projected/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-kube-api-access-jdqm8\") pod \"distinct-w0-4b4c34-kserve-66c49c5849-55dlq\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:34.693883 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:34.693842    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:35.056102 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:35.056071    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"]
Oct 30 19:33:35.056794 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:33:35.056763    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdc58f78_58ef_4981_b1a9_4c6aa8b781fc.slice/crio-cf569081b22d7627213ba9d7437a45775f130c1e05f407bdbdc87254880751be WatchSource:0}: Error finding container cf569081b22d7627213ba9d7437a45775f130c1e05f407bdbdc87254880751be: Status 404 returned error can't find the container with id cf569081b22d7627213ba9d7437a45775f130c1e05f407bdbdc87254880751be
Oct 30 19:33:35.229550 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:35.229515    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:35.230100 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:35.230072    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" event={"ID":"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc","Type":"ContainerStarted","Data":"780f333b11a7f7aec3ab0b2d320191cf3a68df4a8a6b408cc3cc19110fa4ed04"}
Oct 30 19:33:35.230150 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:35.230121    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" event={"ID":"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc","Type":"ContainerStarted","Data":"cf569081b22d7627213ba9d7437a45775f130c1e05f407bdbdc87254880751be"}
Oct 30 19:33:35.273853 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:35.273792    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podStartSLOduration=1.273778947 podStartE2EDuration="1.273778947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:35.272463878 +0000 UTC m=+768.571726008" watchObservedRunningTime="2026-10-30 19:33:35.273778947 +0000 UTC m=+768.573041137"
Oct 30 19:33:35.695086 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:35.695056    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:36.376819 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:33:36.376787    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:33:37.084096 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:37.084063    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:38.696467 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:38.696381    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:40.447507 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:40.447466    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:42.005100 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:42.005034    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:43.604171 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:43.604140    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:44.652414 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.652377    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"]
Oct 30 19:33:44.656209 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.656189    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.681748 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.681721    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"]
Oct 30 19:33:44.694612 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.694585    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:44.707749 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.707726    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:44.712209 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.712188    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-workload-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.712302 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.712225    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpkzg\" (UniqueName: \"kubernetes.io/projected/33d1922a-a6fa-4a8e-8148-174f2ef400af-kube-api-access-jpkzg\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.712302 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.712257    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-data\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.712517 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.712405    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d1922a-a6fa-4a8e-8148-174f2ef400af-istiod-ca-cert\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.712517 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.712456    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-envoy\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.712629 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.712561    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-podinfo\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.712629 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.712604    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-credential-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.712754 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.712641    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-workload-certs\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.712811 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.712777    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-token\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.720739 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.720719    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d797d2-openshift-default-dockercfg-qxcvg\""
Oct 30 19:33:44.813414 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.813389    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-token\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.813523 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.813445    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-workload-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.813523 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.813477    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpkzg\" (UniqueName: \"kubernetes.io/projected/33d1922a-a6fa-4a8e-8148-174f2ef400af-kube-api-access-jpkzg\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.813627 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.813521    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-data\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.813627 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.813547    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d1922a-a6fa-4a8e-8148-174f2ef400af-istiod-ca-cert\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.813720 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.813672    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-envoy\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.813772 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.813738    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-podinfo\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.813834 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.813783    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-credential-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.813893 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.813837    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-workload-certs\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.813989 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.813967    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-workload-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.814151 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.814029    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-data\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.814231 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.814133    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-credential-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.814231 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.814216    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-workload-certs\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.814392 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.814351    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d1922a-a6fa-4a8e-8148-174f2ef400af-istiod-ca-cert\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.815872 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.815853    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-envoy\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.816081 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.816063    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-podinfo\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.833080 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.833050    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-token\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.833244 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.833226    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpkzg\" (UniqueName: \"kubernetes.io/projected/33d1922a-a6fa-4a8e-8148-174f2ef400af-kube-api-access-jpkzg\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:44.971692 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:44.971614    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:45.135119 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:45.135081    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:45.293095 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:45.293069    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:33:45.295545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:45.295516    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"]
Oct 30 19:33:45.295662 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:33:45.295574    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d1922a_a6fa_4a8e_8148_174f2ef400af.slice/crio-bbff620db2db735864a8d080e1b7673ff67e7c2f106b59f7d266e60c02a61f92 WatchSource:0}: Error finding container bbff620db2db735864a8d080e1b7673ff67e7c2f106b59f7d266e60c02a61f92: Status 404 returned error can't find the container with id bbff620db2db735864a8d080e1b7673ff67e7c2f106b59f7d266e60c02a61f92
Oct 30 19:33:45.297829 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:45.297792    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:33:45.297904 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:45.297861    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:33:45.297904 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:45.297890    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:33:46.288317 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:46.288284    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6" event={"ID":"33d1922a-a6fa-4a8e-8148-174f2ef400af","Type":"ContainerStarted","Data":"4d8c5a2733261086e6f340825dd861d3ba0a3b0f5d792e9ce68d3648f56fbf55"}
Oct 30 19:33:46.288695 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:46.288343    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6" event={"ID":"33d1922a-a6fa-4a8e-8148-174f2ef400af","Type":"ContainerStarted","Data":"bbff620db2db735864a8d080e1b7673ff67e7c2f106b59f7d266e60c02a61f92"}
Oct 30 19:33:46.370108 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:46.370048    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6" podStartSLOduration=2.370037082 podStartE2EDuration="2.370037082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:46.358514133 +0000 UTC m=+779.657776263" watchObservedRunningTime="2026-10-30 19:33:46.370037082 +0000 UTC m=+779.669299212"
Oct 30 19:33:46.729392 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:46.725923    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:46.975998 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:46.975965    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:46.989598 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:46.983818    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:33:46.989598 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:46.983948    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6" podUID="33d1922a-a6fa-4a8e-8148-174f2ef400af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 19:33:47.972252 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:47.972221    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:33:47.972755 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:47.972722    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6" podUID="33d1922a-a6fa-4a8e-8148-174f2ef400af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 19:33:48.973697 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:48.973655    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:33:48.974211 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:48.973729    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6" podUID="33d1922a-a6fa-4a8e-8148-174f2ef400af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 19:33:49.499411 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:49.499100    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:49.972654 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:49.972619    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:33:49.972824 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:49.972677    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6" podUID="33d1922a-a6fa-4a8e-8148-174f2ef400af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 19:33:50.876213 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:50.876164    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:50.972762 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:50.972729    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 19:33:50.972902 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:50.972777    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6" podUID="33d1922a-a6fa-4a8e-8148-174f2ef400af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 19:33:52.329868 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:52.329830    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:52.853241 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:52.853207    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:52.853619 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:52.853601    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:52.854504 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:52.854479    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:33:53.786798 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:53.786762    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:54.094834 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.094761    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:54.400945 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.400905    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"]
Oct 30 19:33:54.404988 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.404965    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.438218 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.438178    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f913c6\"/\"kube-root-ca.crt\""
Oct 30 19:33:54.438334 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.438240    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f913c6\"/\"openshift-service-ca.crt\""
Oct 30 19:33:54.438334 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.438240    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f913c6\"/\"distinct-w4-f913c6-kserve-self-signed-certs\""
Oct 30 19:33:54.438334 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.438273    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f913c6\"/\"default-dockercfg-bcjnr\""
Oct 30 19:33:54.443277 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.443251    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"]
Oct 30 19:33:54.472426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.472404    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-model-cache\") pod \"distinct-w4-f913c6-kserve-7767c75f85-7pghc\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") " pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.472540 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.472434    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-tls-certs\") pod \"distinct-w4-f913c6-kserve-7767c75f85-7pghc\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") " pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.472540 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.472473    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-dshm\") pod \"distinct-w4-f913c6-kserve-7767c75f85-7pghc\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") " pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.472927 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.472904    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp42w\" (UniqueName: \"kubernetes.io/projected/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-kube-api-access-fp42w\") pod \"distinct-w4-f913c6-kserve-7767c75f85-7pghc\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") " pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.473007 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.472961    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-home\") pod \"distinct-w4-f913c6-kserve-7767c75f85-7pghc\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") " pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.473105 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.473086    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-tmp-dir\") pod \"distinct-w4-f913c6-kserve-7767c75f85-7pghc\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") " pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.510069 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.510041    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:54.574343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.574321    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-home\") pod \"distinct-w4-f913c6-kserve-7767c75f85-7pghc\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") " pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.574454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.574391    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-tmp-dir\") pod \"distinct-w4-f913c6-kserve-7767c75f85-7pghc\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") " pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.574513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.574497    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-model-cache\") pod \"distinct-w4-f913c6-kserve-7767c75f85-7pghc\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") " pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.574570 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.574524    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-tls-certs\") pod \"distinct-w4-f913c6-kserve-7767c75f85-7pghc\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") " pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.574570 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.574561    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-dshm\") pod \"distinct-w4-f913c6-kserve-7767c75f85-7pghc\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") " pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.574670 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.574614    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp42w\" (UniqueName: \"kubernetes.io/projected/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-kube-api-access-fp42w\") pod \"distinct-w4-f913c6-kserve-7767c75f85-7pghc\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") " pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.574842 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.574819    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-tmp-dir\") pod \"distinct-w4-f913c6-kserve-7767c75f85-7pghc\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") " pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.574958 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.574887    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-model-cache\") pod \"distinct-w4-f913c6-kserve-7767c75f85-7pghc\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") " pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.575004 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.574956    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-home\") pod \"distinct-w4-f913c6-kserve-7767c75f85-7pghc\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") " pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.577056 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.577031    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-dshm\") pod \"distinct-w4-f913c6-kserve-7767c75f85-7pghc\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") " pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.577192 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.577170    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-tls-certs\") pod \"distinct-w4-f913c6-kserve-7767c75f85-7pghc\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") " pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.596691 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.596662    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp42w\" (UniqueName: \"kubernetes.io/projected/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-kube-api-access-fp42w\") pod \"distinct-w4-f913c6-kserve-7767c75f85-7pghc\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") " pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:54.729436 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:54.729345    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:55.026094 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:55.026064    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"]
Oct 30 19:33:55.026562 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:33:55.026141    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e3a8512_59ff_45bf_b7c7_55369c05d8ce.slice/crio-0666501b4bc96f8570097fb7c1e5ff15c87f55b9c7686caaef54b0d52946f06c WatchSource:0}: Error finding container 0666501b4bc96f8570097fb7c1e5ff15c87f55b9c7686caaef54b0d52946f06c: Status 404 returned error can't find the container with id 0666501b4bc96f8570097fb7c1e5ff15c87f55b9c7686caaef54b0d52946f06c
Oct 30 19:33:55.434155 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:55.434119    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" event={"ID":"5e3a8512-59ff-45bf-b7c7-55369c05d8ce","Type":"ContainerStarted","Data":"3979c4a006abcd517ebee57e8953feae98b86d3ecd048830bc7569f4a745c51a"}
Oct 30 19:33:55.434315 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:55.434167    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" event={"ID":"5e3a8512-59ff-45bf-b7c7-55369c05d8ce","Type":"ContainerStarted","Data":"0666501b4bc96f8570097fb7c1e5ff15c87f55b9c7686caaef54b0d52946f06c"}
Oct 30 19:33:55.479538 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:55.479488    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podStartSLOduration=1.479476028 podStartE2EDuration="1.479476028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:55.475930208 +0000 UTC m=+788.775192336" watchObservedRunningTime="2026-10-30 19:33:55.479476028 +0000 UTC m=+788.778738158"
Oct 30 19:33:55.543995 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:55.543963    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:55.729759 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:55.729683    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:33:57.216208 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:57.216174    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:58.715445 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:58.715414    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:33:58.874127 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:33:58.874097    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:00.307290 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:00.307258    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:02.066536 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:02.066503    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:04.019848 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:04.019818    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:04.729845 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:04.729819    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:34:04.742477 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:04.742453    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:34:05.494138 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:05.494112    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:34:05.604020 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:05.603968    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:06.372888 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:34:06.372851    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:34:08.240057 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:08.240021    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:09.661627 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:09.661597    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:11.067336 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:11.067300    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:12.611726 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.611684    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:12.733182 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.733149    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"]
Oct 30 19:34:12.737021 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.736999    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:12.758163 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.758143    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"]
Oct 30 19:34:12.767632 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.767612    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd23d1\"/\"distinct-w3-dd23d1-kserve-self-signed-certs\""
Oct 30 19:34:12.824719 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.824690    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:12.880381 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.880334    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-tmp-dir\") pod \"distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:12.880481 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.880409    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d2s9\" (UniqueName: \"kubernetes.io/projected/bbf52b02-27b0-485c-9e4d-9a804a8853db-kube-api-access-6d2s9\") pod \"distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:12.880526 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.880516    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-home\") pod \"distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:12.880562 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.880537    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-dshm\") pod \"distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:12.880596 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.880560    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf52b02-27b0-485c-9e4d-9a804a8853db-tls-certs\") pod \"distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:12.880653 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.880632    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-model-cache\") pod \"distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:12.982248 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.982211    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-tmp-dir\") pod \"distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:12.982480 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.982261    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d2s9\" (UniqueName: \"kubernetes.io/projected/bbf52b02-27b0-485c-9e4d-9a804a8853db-kube-api-access-6d2s9\") pod \"distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:12.982480 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.982393    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-home\") pod \"distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:12.982480 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.982426    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-dshm\") pod \"distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:12.982480 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.982458    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf52b02-27b0-485c-9e4d-9a804a8853db-tls-certs\") pod \"distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:12.982682 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.982499    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-model-cache\") pod \"distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:12.982682 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.982634    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-tmp-dir\") pod \"distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:12.982784 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.982764    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-home\") pod \"distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:12.982989 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.982970    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-model-cache\") pod \"distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:12.984618 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.984594    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-dshm\") pod \"distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:12.984858 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:12.984840    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf52b02-27b0-485c-9e4d-9a804a8853db-tls-certs\") pod \"distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:13.002384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:13.002341    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d2s9\" (UniqueName: \"kubernetes.io/projected/bbf52b02-27b0-485c-9e4d-9a804a8853db-kube-api-access-6d2s9\") pod \"distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") " pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:13.049580 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:13.049556    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:13.345432 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:13.345399    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"]
Oct 30 19:34:13.345644 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:34:13.345623    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbf52b02_27b0_485c_9e4d_9a804a8853db.slice/crio-38d16b750ae07c9eafe757be5ba7dc4e352426d370351e1c8389afcf8e3753b8 WatchSource:0}: Error finding container 38d16b750ae07c9eafe757be5ba7dc4e352426d370351e1c8389afcf8e3753b8: Status 404 returned error can't find the container with id 38d16b750ae07c9eafe757be5ba7dc4e352426d370351e1c8389afcf8e3753b8
Oct 30 19:34:13.526057 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:13.526025    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" event={"ID":"bbf52b02-27b0-485c-9e4d-9a804a8853db","Type":"ContainerStarted","Data":"7e72ca4902d11d9ded49d237498d67736afc4e50b614b93632eebd49dbada287"}
Oct 30 19:34:13.526057 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:13.526065    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" event={"ID":"bbf52b02-27b0-485c-9e4d-9a804a8853db","Type":"ContainerStarted","Data":"38d16b750ae07c9eafe757be5ba7dc4e352426d370351e1c8389afcf8e3753b8"}
Oct 30 19:34:13.569733 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:13.569685    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podStartSLOduration=1.569672201 podStartE2EDuration="1.569672201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:13.567561921 +0000 UTC m=+806.866824049" watchObservedRunningTime="2026-10-30 19:34:13.569672201 +0000 UTC m=+806.868934335"
Oct 30 19:34:14.050438 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:14.050347    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:14.115979 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:14.115944    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:14.860011 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:14.859983    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"]
Oct 30 19:34:14.863668 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:14.863651    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:14.887255 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:14.887231    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"]
Oct 30 19:34:15.001248 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.001214    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"]
Oct 30 19:34:15.005002 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.004980    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.005341 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.005324    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-envoy\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.005434 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.005409    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istiod-ca-cert\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.005491 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.005451    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-credential-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.005592 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.005571    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-workload-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.005685 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.005607    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-podinfo\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.005793 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.005772    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-workload-certs\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.005846 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.005819    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8l7m\" (UniqueName: \"kubernetes.io/projected/15c32bf5-e05e-41bd-8482-22325ad0bf0c-kube-api-access-p8l7m\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.005885 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.005874    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-data\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.005996 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.005957    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-token\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.024199 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.024176    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-3735b2-openshift-default-dockercfg-z8mrd\""
Oct 30 19:34:15.027060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.027034    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"]
Oct 30 19:34:15.106952 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.106922    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.106952 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.106954    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.107343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.106984    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-envoy\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.107343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107079    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istiod-ca-cert\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.107343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107109    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-credential-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.107343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107159    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.107343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107178    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-workload-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.107343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107200    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.107343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107265    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-podinfo\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.107343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107338    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.107810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107435    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-workload-certs\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.107810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107464    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbk8s\" (UniqueName: \"kubernetes.io/projected/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-kube-api-access-cbk8s\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.107810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107528    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8l7m\" (UniqueName: \"kubernetes.io/projected/15c32bf5-e05e-41bd-8482-22325ad0bf0c-kube-api-access-p8l7m\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.107810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107586    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.107810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107614    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-credential-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.107810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107618    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.107810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107690    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-data\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.107810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107757    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-token\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.107810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107798    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istiod-ca-cert\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.107810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107803    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.108335 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107825    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-workload-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.108335 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.107983    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-data\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.108335 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.108195    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-workload-certs\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.109189 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.109169    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-envoy\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.110002 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.109986    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-podinfo\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.126876 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.126846    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8l7m\" (UniqueName: \"kubernetes.io/projected/15c32bf5-e05e-41bd-8482-22325ad0bf0c-kube-api-access-p8l7m\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.126968 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.126942    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-token\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.177651 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.177632    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:15.209388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.209347    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.209508 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.209434    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.209508 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.209465    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.209636 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.209546    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.209696 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.209674    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.209757 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.209743    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.210117 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.209820    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.210117 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.209831    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbk8s\" (UniqueName: \"kubernetes.io/projected/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-kube-api-access-cbk8s\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.210117 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.209859    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.210117 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.209918    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.210117 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.209927    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.210117 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.209947    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.210411 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.210131    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.210411 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.210179    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.212103 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.212083    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.212226 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.212210    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.230710 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.230304    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.231090 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.231045    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbk8s\" (UniqueName: \"kubernetes.io/projected/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-kube-api-access-cbk8s\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.318474 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.318448    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:15.494601 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.494576    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"]
Oct 30 19:34:15.495810 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:34:15.495784    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c32bf5_e05e_41bd_8482_22325ad0bf0c.slice/crio-68b9ed59cd0c4ff98a128dadda1a37ed31876b26c406edee7a160279ee114c53 WatchSource:0}: Error finding container 68b9ed59cd0c4ff98a128dadda1a37ed31876b26c406edee7a160279ee114c53: Status 404 returned error can't find the container with id 68b9ed59cd0c4ff98a128dadda1a37ed31876b26c406edee7a160279ee114c53
Oct 30 19:34:15.497611 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.497576    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:34:15.497680 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.497642    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:34:15.497680 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.497674    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:34:15.544928 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.544896    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr" event={"ID":"15c32bf5-e05e-41bd-8482-22325ad0bf0c","Type":"ContainerStarted","Data":"68b9ed59cd0c4ff98a128dadda1a37ed31876b26c406edee7a160279ee114c53"}
Oct 30 19:34:15.630444 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:34:15.630415    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced985e1_6978_4dcb_8cc9_7312fe9a16d7.slice/crio-011134bd88cc580852104fbd7544641098b416fd0013888ea4f4e3f684043bbe WatchSource:0}: Error finding container 011134bd88cc580852104fbd7544641098b416fd0013888ea4f4e3f684043bbe: Status 404 returned error can't find the container with id 011134bd88cc580852104fbd7544641098b416fd0013888ea4f4e3f684043bbe
Oct 30 19:34:15.630635 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.630618    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"]
Oct 30 19:34:15.633283 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.633247    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:34:15.633463 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.633315    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:34:15.633463 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.633346    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:34:15.651955 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:15.651923    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:16.560036 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:16.560002    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f" event={"ID":"ced985e1-6978-4dcb-8cc9-7312fe9a16d7","Type":"ContainerStarted","Data":"f9ee5b84849cb98b147cbc30f4455e3b292ab56a2090571e8a5f345ee05072b2"}
Oct 30 19:34:16.560527 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:16.560058    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f" event={"ID":"ced985e1-6978-4dcb-8cc9-7312fe9a16d7","Type":"ContainerStarted","Data":"011134bd88cc580852104fbd7544641098b416fd0013888ea4f4e3f684043bbe"}
Oct 30 19:34:16.561637 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:16.561609    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr" event={"ID":"15c32bf5-e05e-41bd-8482-22325ad0bf0c","Type":"ContainerStarted","Data":"27e62fca93e5cfdc95a6f7124756da1b984ce0803060b2ce6a06400da6451c32"}
Oct 30 19:34:16.603985 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:16.603937    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f" podStartSLOduration=2.6039197229999997 podStartE2EDuration="2.603919723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:16.603918274 +0000 UTC m=+809.903180405" watchObservedRunningTime="2026-10-30 19:34:16.603919723 +0000 UTC m=+809.903181853"
Oct 30 19:34:16.655930 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:16.655872    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr" podStartSLOduration=2.6558562439999998 podStartE2EDuration="2.655856244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:16.653868926 +0000 UTC m=+809.953131057" watchObservedRunningTime="2026-10-30 19:34:16.655856244 +0000 UTC m=+809.955118373"
Oct 30 19:34:17.177785 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:17.177754    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:17.177989 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:17.177957    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:17.179123 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:17.179092    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:34:17.179169 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:17.179145    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr" podUID="15c32bf5-e05e-41bd-8482-22325ad0bf0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 19:34:17.325384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:17.321689    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:17.347529 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:17.347496    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:34:17.347813 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:17.347764    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f" podUID="ced985e1-6978-4dcb-8cc9-7312fe9a16d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 19:34:18.179000 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:18.178962    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:34:18.179419 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:18.179015    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr" podUID="15c32bf5-e05e-41bd-8482-22325ad0bf0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 19:34:18.327386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:18.323750    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:34:18.327386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:18.323809    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f" podUID="ced985e1-6978-4dcb-8cc9-7312fe9a16d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 19:34:18.698503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:18.698467    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:18.853386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:18.848419    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"]
Oct 30 19:34:18.870621 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:18.870582    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:18.883884 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:18.883853    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"]
Oct 30 19:34:18.915384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:18.904453    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b4c34\"/\"distinct-2-w0-4b4c34-kserve-self-signed-certs\""
Oct 30 19:34:18.946381 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:18.938802    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:18.982181 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:18.982114    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-dshm\") pod \"distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:18.982181 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:18.982171    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-tmp-dir\") pod \"distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:18.982724 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:18.982703    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-home\") pod \"distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:18.982859 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:18.982843    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kknxw\" (UniqueName: \"kubernetes.io/projected/aa037781-1c62-492d-adcc-006600af37ff-kube-api-access-kknxw\") pod \"distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:18.982973 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:18.982954    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-model-cache\") pod \"distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:18.983039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:18.982980    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa037781-1c62-492d-adcc-006600af37ff-tls-certs\") pod \"distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:19.083920 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.083875    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-home\") pod \"distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:19.084147 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.084106    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kknxw\" (UniqueName: \"kubernetes.io/projected/aa037781-1c62-492d-adcc-006600af37ff-kube-api-access-kknxw\") pod \"distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:19.084233 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.084169    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-model-cache\") pod \"distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:19.084233 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.084209    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa037781-1c62-492d-adcc-006600af37ff-tls-certs\") pod \"distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:19.084348 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.084320    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-dshm\") pod \"distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:19.084505 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.084484    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-tmp-dir\") pod \"distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:19.084644 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.084620    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-home\") pod \"distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:19.084817 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.084791    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-tmp-dir\") pod \"distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:19.085443 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.085421    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-model-cache\") pod \"distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:19.086899 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.086877    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-dshm\") pod \"distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:19.087383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.087335    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa037781-1c62-492d-adcc-006600af37ff-tls-certs\") pod \"distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:19.121385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.104942    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kknxw\" (UniqueName: \"kubernetes.io/projected/aa037781-1c62-492d-adcc-006600af37ff-kube-api-access-kknxw\") pod \"distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") " pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:19.178864 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.178835    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:34:19.179072 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.179050    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr" podUID="15c32bf5-e05e-41bd-8482-22325ad0bf0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 19:34:19.190635 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.190613    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:19.362385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.361500    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:34:19.362385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.361563    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f" podUID="ced985e1-6978-4dcb-8cc9-7312fe9a16d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 19:34:19.558245 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.558217    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"]
Oct 30 19:34:19.560307 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:34:19.560286    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa037781_1c62_492d_adcc_006600af37ff.slice/crio-5dee4b07702292c330a748c509dfe0bb09d832d820c3fad9c9b058d73356d19f WatchSource:0}: Error finding container 5dee4b07702292c330a748c509dfe0bb09d832d820c3fad9c9b058d73356d19f: Status 404 returned error can't find the container with id 5dee4b07702292c330a748c509dfe0bb09d832d820c3fad9c9b058d73356d19f
Oct 30 19:34:19.581419 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:19.581396    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" event={"ID":"aa037781-1c62-492d-adcc-006600af37ff","Type":"ContainerStarted","Data":"5dee4b07702292c330a748c509dfe0bb09d832d820c3fad9c9b058d73356d19f"}
Oct 30 19:34:20.178299 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:20.178266    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:34:20.178502 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:20.178317    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr" podUID="15c32bf5-e05e-41bd-8482-22325ad0bf0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 19:34:20.320384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:20.319740    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:34:20.320384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:20.319853    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f" podUID="ced985e1-6978-4dcb-8cc9-7312fe9a16d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 19:34:20.355397 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:20.352663    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:20.591596 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:20.591506    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" event={"ID":"aa037781-1c62-492d-adcc-006600af37ff","Type":"ContainerStarted","Data":"463c5b2a01fe2bb2d5910e2addaf577df50697445bbbdeabe50ed4b2dac3a2d6"}
Oct 30 19:34:20.652385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:20.650404    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podStartSLOduration=2.650352337 podStartE2EDuration="2.650352337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:20.646745582 +0000 UTC m=+813.946007712" watchObservedRunningTime="2026-10-30 19:34:20.650352337 +0000 UTC m=+813.949614467"
Oct 30 19:34:21.178853 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:21.178816    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 19:34:21.179035 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:21.178873    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr" podUID="15c32bf5-e05e-41bd-8482-22325ad0bf0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 19:34:21.191048 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:21.191021    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:21.321975 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:21.321939    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:34:21.322579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:21.322545    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f" podUID="ced985e1-6978-4dcb-8cc9-7312fe9a16d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 19:34:22.047669 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:22.047620    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:22.298347 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:22.298011    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:22.299328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:22.299305    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:22.300045 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:22.300027    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:34:22.319756 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:22.319734    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:34:22.319838 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:22.319779    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f" podUID="ced985e1-6978-4dcb-8cc9-7312fe9a16d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 19:34:23.049950 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:23.049912    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:23.062221 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:23.062191    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:23.323716 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:23.323651    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:23.324016 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:23.323999    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:23.324579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:23.324561    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:34:23.598304 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:23.598170    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:23.632592 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:23.632564    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:34:23.817883 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:23.817844    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:25.185707 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:25.185675    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:26.702423 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:26.702389    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:28.903044 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:28.903013    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:29.191216 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.191141    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:29.203788 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.203759    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:29.665014 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.664982    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:34:29.746898 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.746856    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"]
Oct 30 19:34:29.750849 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.750827    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.773594 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.773562    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"]
Oct 30 19:34:29.850132 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.850103    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-data\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.850329 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.850153    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-envoy\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.850329 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.850179    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-token\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.850329 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.850246    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2fa6bc2-624f-4017-898f-2317c9064c73-istiod-ca-cert\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.850329 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.850293    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-workload-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.850557 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.850423    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-credential-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.850557 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.850460    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-podinfo\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.850638 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.850570    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-workload-certs\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.850638 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.850595    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmgd9\" (UniqueName: \"kubernetes.io/projected/d2fa6bc2-624f-4017-898f-2317c9064c73-kube-api-access-kmgd9\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.951630 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.951552    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-workload-certs\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.951630 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.951590    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmgd9\" (UniqueName: \"kubernetes.io/projected/d2fa6bc2-624f-4017-898f-2317c9064c73-kube-api-access-kmgd9\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.952130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.951667    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-data\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.952130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.951718    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-envoy\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.952130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.951836    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-token\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.952130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.951881    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2fa6bc2-624f-4017-898f-2317c9064c73-istiod-ca-cert\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.952130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.951925    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-workload-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.952130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.951995    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-credential-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.952130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.952016    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-workload-certs\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.952130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.952066    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-podinfo\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.952603 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.952143    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-data\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.952603 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.952342    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-workload-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.952603 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.952513    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-credential-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.952712 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.952611    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2fa6bc2-624f-4017-898f-2317c9064c73-istiod-ca-cert\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.953965 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.953942    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-envoy\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.954398 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.954381    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-podinfo\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.971321 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.971299    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-token\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:29.971567 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:29.971549    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmgd9\" (UniqueName: \"kubernetes.io/projected/d2fa6bc2-624f-4017-898f-2317c9064c73-kube-api-access-kmgd9\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:30.071853 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:30.071830    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:30.379748 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:34:30.379718    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2fa6bc2_624f_4017_898f_2317c9064c73.slice/crio-f231de1d36dd03f857fd19b3f29c2406119179a9fc0afdb62ba712e1b8f5f86d WatchSource:0}: Error finding container f231de1d36dd03f857fd19b3f29c2406119179a9fc0afdb62ba712e1b8f5f86d: Status 404 returned error can't find the container with id f231de1d36dd03f857fd19b3f29c2406119179a9fc0afdb62ba712e1b8f5f86d
Oct 30 19:34:30.379863 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:30.379831    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"]
Oct 30 19:34:30.382163 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:30.382128    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:34:30.382260 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:30.382193    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:34:30.382260 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:30.382224    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:34:30.409030 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:30.408961    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:30.662225 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:30.662137    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7" event={"ID":"d2fa6bc2-624f-4017-898f-2317c9064c73","Type":"ContainerStarted","Data":"205793fcae347b200b1755d5f49f4eeafaefc3aeedb68e7b2d547b0381b46886"}
Oct 30 19:34:30.662225 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:30.662193    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7" event={"ID":"d2fa6bc2-624f-4017-898f-2317c9064c73","Type":"ContainerStarted","Data":"f231de1d36dd03f857fd19b3f29c2406119179a9fc0afdb62ba712e1b8f5f86d"}
Oct 30 19:34:30.706322 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:30.706259    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7" podStartSLOduration=1.7062396149999999 podStartE2EDuration="1.706239615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:30.703072791 +0000 UTC m=+824.002334920" watchObservedRunningTime="2026-10-30 19:34:30.706239615 +0000 UTC m=+824.005501744"
Oct 30 19:34:31.052478 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:31.052389    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:31.072444 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:31.072423    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:31.074104 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:31.074077    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:34:31.074222 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:31.074149    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7" podUID="d2fa6bc2-624f-4017-898f-2317c9064c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:34:32.072617 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:32.072583    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:34:32.073121 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:32.072635    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7" podUID="d2fa6bc2-624f-4017-898f-2317c9064c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:34:32.186963 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:32.186922    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:33.072664 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:33.072628    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:34:33.073069 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:33.072681    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7" podUID="d2fa6bc2-624f-4017-898f-2317c9064c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:34:33.936268 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:33.936230    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:34.072954 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:34.072922    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:34:34.073332 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:34.072985    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7" podUID="d2fa6bc2-624f-4017-898f-2317c9064c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:34:35.072885 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:35.072851    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:34:35.073059 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:35.072925    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7" podUID="d2fa6bc2-624f-4017-898f-2317c9064c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:34:35.728902 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:35.728852    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:36.072342 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.072268    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:34:36.072342 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.072319    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7" podUID="d2fa6bc2-624f-4017-898f-2317c9064c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:34:36.373841 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:34:36.373801    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:34:36.685974 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.685894    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"]
Oct 30 19:34:36.707634 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.707593    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:36.710922 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.710889    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"]
Oct 30 19:34:36.723863 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.723839    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ef9a5a\"/\"distinct-w6-ef9a5a-kserve-self-signed-certs\""
Oct 30 19:34:36.735479 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.735457    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ef9a5a\"/\"default-dockercfg-wtrg6\""
Oct 30 19:34:36.735588 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.735505    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ef9a5a\"/\"openshift-service-ca.crt\""
Oct 30 19:34:36.735588 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.735548    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ef9a5a\"/\"kube-root-ca.crt\""
Oct 30 19:34:36.782739 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.782698    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:36.854155 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.854126    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-tmp-dir\") pod \"distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") " pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:36.854155 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.854156    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpscf\" (UniqueName: \"kubernetes.io/projected/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-kube-api-access-dpscf\") pod \"distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") " pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:36.854412 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.854257    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-model-cache\") pod \"distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") " pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:36.854412 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.854377    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-dshm\") pod \"distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") " pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:36.854539 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.854438    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-tls-certs\") pod \"distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") " pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:36.854647 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.854630    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-home\") pod \"distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") " pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:36.956107 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.955994    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-home\") pod \"distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") " pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:36.956107 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.956073    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-tmp-dir\") pod \"distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") " pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:36.956107 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.956100    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpscf\" (UniqueName: \"kubernetes.io/projected/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-kube-api-access-dpscf\") pod \"distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") " pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:36.956416 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.956189    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-model-cache\") pod \"distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") " pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:36.956416 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.956236    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-dshm\") pod \"distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") " pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:36.956416 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.956274    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-tls-certs\") pod \"distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") " pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:36.957097 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.957065    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-tmp-dir\") pod \"distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") " pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:36.957097 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.957092    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-home\") pod \"distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") " pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:36.957987 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.957857    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-model-cache\") pod \"distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") " pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:36.959803 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.959769    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-tls-certs\") pod \"distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") " pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:36.959986 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.959967    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-dshm\") pod \"distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") " pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:36.976671 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:36.976644    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpscf\" (UniqueName: \"kubernetes.io/projected/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-kube-api-access-dpscf\") pod \"distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") " pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:37.023411 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:37.023375    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:37.123960 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:37.123930    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:37.124896 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:37.124819    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:37.125557 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:37.125529    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:34:37.343807 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:37.343778    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"]
Oct 30 19:34:37.568947 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:37.568909    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:37.710548 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:37.710512    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" event={"ID":"5a486ae6-e874-4317-ab9b-e6cdebc7ce89","Type":"ContainerStarted","Data":"60bc701d79e0d6bffad3395b78027803f54bc7c04a343fbda6448086657b520c"}
Oct 30 19:34:37.710724 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:37.710556    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" event={"ID":"5a486ae6-e874-4317-ab9b-e6cdebc7ce89","Type":"ContainerStarted","Data":"e84435733914f07e72b02865e077aca4d635ffd27dd3d69fef6a5a67f37321de"}
Oct 30 19:34:37.753889 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:37.753831    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podStartSLOduration=1.753811086 podStartE2EDuration="1.753811086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:37.75066672 +0000 UTC m=+831.049928851" watchObservedRunningTime="2026-10-30 19:34:37.753811086 +0000 UTC m=+831.053073195"
Oct 30 19:34:38.024101 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:38.024019    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:39.118570 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:39.118535    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:40.588847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:40.588808    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:42.077880 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:42.077838    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:43.687726 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:43.687666    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:45.195883 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:45.195834    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:47.024323 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:47.024293    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:47.037148 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:47.037126    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:47.246689 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:47.246643    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:47.813379 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:47.813328    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:34:48.609545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:48.609496    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:48.752579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:48.752546    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:50.331498 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:50.331417    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:51.980944 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:51.980903    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:52.371520 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:52.371482    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:53.535379 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.535322    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:53.719437 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.719405    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"]
Oct 30 19:34:53.725018 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.724994    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:53.742869 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.742842    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"]
Oct 30 19:34:53.750819 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.750779    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3735b2\"/\"distinct-2-w2-3735b2-kserve-self-signed-certs\""
Oct 30 19:34:53.839857 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.839830    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:53.867811 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.867788    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-home\") pod \"distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:53.867899 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.867825    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs7m6\" (UniqueName: \"kubernetes.io/projected/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-kube-api-access-zs7m6\") pod \"distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:53.867948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.867902    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-tls-certs\") pod \"distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:53.867989 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.867956    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-tmp-dir\") pod \"distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:53.868097 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.868079    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-dshm\") pod \"distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:53.868132 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.868113    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-model-cache\") pod \"distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:53.969867 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.969820    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-tls-certs\") pod \"distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:53.970061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.969899    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-tmp-dir\") pod \"distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:53.970061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.970033    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-dshm\") pod \"distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:53.970175 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.970061    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-model-cache\") pod \"distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:53.970175 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.970150    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-home\") pod \"distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:53.970292 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.970214    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs7m6\" (UniqueName: \"kubernetes.io/projected/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-kube-api-access-zs7m6\") pod \"distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:53.970483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.970455    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-tmp-dir\") pod \"distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:53.970765 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.970734    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-home\") pod \"distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:53.971009 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.970987    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-model-cache\") pod \"distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:53.972552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.972484    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-dshm\") pod \"distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:53.973786 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.973761    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-tls-certs\") pod \"distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:53.990715 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:53.990686    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs7m6\" (UniqueName: \"kubernetes.io/projected/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-kube-api-access-zs7m6\") pod \"distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") " pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:54.072413 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:54.072315    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:54.366179 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:54.366147    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"]
Oct 30 19:34:54.366310 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:34:54.366221    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d5b13af_b0c4_4343_80ca_5c5e0d0c769f.slice/crio-6ac4aa851899c1124e77db67f5aced447f46d7fefe038b963863747e47222893 WatchSource:0}: Error finding container 6ac4aa851899c1124e77db67f5aced447f46d7fefe038b963863747e47222893: Status 404 returned error can't find the container with id 6ac4aa851899c1124e77db67f5aced447f46d7fefe038b963863747e47222893
Oct 30 19:34:54.851911 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:54.851878    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" event={"ID":"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f","Type":"ContainerStarted","Data":"c7884fb36ae79ee5584ddbe62cedf8720bc522853580c91b186f2dfdd76498ae"}
Oct 30 19:34:54.852284 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:54.851925    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" event={"ID":"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f","Type":"ContainerStarted","Data":"6ac4aa851899c1124e77db67f5aced447f46d7fefe038b963863747e47222893"}
Oct 30 19:34:54.887133 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:54.887082    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podStartSLOduration=1.887069796 podStartE2EDuration="1.887069796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:54.88547849 +0000 UTC m=+848.184740620" watchObservedRunningTime="2026-10-30 19:34:54.887069796 +0000 UTC m=+848.186331924"
Oct 30 19:34:55.072635 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:55.072605    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:34:55.079614 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:55.079576    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:56.482317 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:56.482270    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:58.089565 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:58.089473    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:34:59.892596 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:34:59.892518    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:00.487840 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.487795    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:00.546337 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.546303    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"]
Oct 30 19:35:00.550670 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.550647    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.570115 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.570085    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-dd23d1-openshift-default-dockercfg-qvtg4\""
Oct 30 19:35:00.609487 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.609449    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"]
Oct 30 19:35:00.647358 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.647320    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpvk2\" (UniqueName: \"kubernetes.io/projected/bcdfb861-99cb-4c80-882f-fed7871177c2-kube-api-access-rpvk2\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.647627 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.647418    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-workload-certs\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.647627 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.647508    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-podinfo\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.647817 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.647652    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-workload-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.647817 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.647729    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcdfb861-99cb-4c80-882f-fed7871177c2-istiod-ca-cert\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.647817 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.647795    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-envoy\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.648023 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.647844    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-token\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.648023 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.647887    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-data\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.648023 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.647926    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-credential-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.748715 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.748633    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-envoy\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.748715 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.748689    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-token\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.748956 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.748731    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-data\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.748956 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.748768    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-credential-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.748956 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.748902    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpvk2\" (UniqueName: \"kubernetes.io/projected/bcdfb861-99cb-4c80-882f-fed7871177c2-kube-api-access-rpvk2\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.749130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.748984    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-workload-certs\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.749130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.749009    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-podinfo\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.749130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.749038    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-workload-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.749130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.749075    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcdfb861-99cb-4c80-882f-fed7871177c2-istiod-ca-cert\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.749948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.749343    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-credential-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.749948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.749411    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-data\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.749948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.749610    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-workload-certs\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.749948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.749901    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcdfb861-99cb-4c80-882f-fed7871177c2-istiod-ca-cert\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.750209 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.750018    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-workload-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.751564 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.751543    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-envoy\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.752238 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.752217    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-podinfo\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.768595 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.768575    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-token\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.769029 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.769011    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpvk2\" (UniqueName: \"kubernetes.io/projected/bcdfb861-99cb-4c80-882f-fed7871177c2-kube-api-access-rpvk2\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.841072 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.841047    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"]
Oct 30 19:35:00.846153 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.846128    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:00.871086 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.871062    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"]
Oct 30 19:35:00.938112 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.938082    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:00.951949 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.951915    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-istio-data\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:00.952069 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.951968    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42358a7-0f5b-476a-831a-cefabde3984e-istiod-ca-cert\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:00.952069 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.952012    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdv4k\" (UniqueName: \"kubernetes.io/projected/e42358a7-0f5b-476a-831a-cefabde3984e-kube-api-access-tdv4k\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:00.952185 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.952145    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42358a7-0f5b-476a-831a-cefabde3984e-istio-podinfo\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:00.952185 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.952172    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-workload-certs\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:00.952287 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.952264    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42358a7-0f5b-476a-831a-cefabde3984e-istio-token\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:00.952345 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.952307    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-istio-envoy\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:00.952440 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.952421    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-workload-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:00.952496 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:00.952452    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-credential-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.054285 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.054205    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42358a7-0f5b-476a-831a-cefabde3984e-istiod-ca-cert\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.054285 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.054256    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdv4k\" (UniqueName: \"kubernetes.io/projected/e42358a7-0f5b-476a-831a-cefabde3984e-kube-api-access-tdv4k\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.054533 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.054346    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42358a7-0f5b-476a-831a-cefabde3984e-istio-podinfo\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.054533 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.054396    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-workload-certs\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.054533 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.054457    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42358a7-0f5b-476a-831a-cefabde3984e-istio-token\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.054533 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.054507    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-istio-envoy\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.054778 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.054574    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-workload-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.054778 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.054598    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-credential-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.054778 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.054722    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-istio-data\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.054938 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.054790    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-workload-certs\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.055057 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.055011    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-istio-data\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.055057 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.055033    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42358a7-0f5b-476a-831a-cefabde3984e-istiod-ca-cert\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.055198 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.055177    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-workload-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.055257 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.055243    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-credential-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.056855 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.056833    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-istio-envoy\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.057278 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.057255    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42358a7-0f5b-476a-831a-cefabde3984e-istio-podinfo\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.073356 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.073335    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42358a7-0f5b-476a-831a-cefabde3984e-istio-token\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.073476 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.073442    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdv4k\" (UniqueName: \"kubernetes.io/projected/e42358a7-0f5b-476a-831a-cefabde3984e-kube-api-access-tdv4k\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.160315 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.160288    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:01.249160 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:35:01.249111    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcdfb861_99cb_4c80_882f_fed7871177c2.slice/crio-7d3a9b178997a540c8a0f786973b3c8705f20befd960de09f5c89d284e05d31b WatchSource:0}: Error finding container 7d3a9b178997a540c8a0f786973b3c8705f20befd960de09f5c89d284e05d31b: Status 404 returned error can't find the container with id 7d3a9b178997a540c8a0f786973b3c8705f20befd960de09f5c89d284e05d31b
Oct 30 19:35:01.250248 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.250225    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"]
Oct 30 19:35:01.251876 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.251842    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:01.251981 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.251922    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:01.251981 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.251950    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:01.460576 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.460552    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"]
Oct 30 19:35:01.461564 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:35:01.461538    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42358a7_0f5b_476a_831a_cefabde3984e.slice/crio-fb4a421533d0a87e448c660530238ab1b0439603f8d5fdfda251fc96c91d7972 WatchSource:0}: Error finding container fb4a421533d0a87e448c660530238ab1b0439603f8d5fdfda251fc96c91d7972: Status 404 returned error can't find the container with id fb4a421533d0a87e448c660530238ab1b0439603f8d5fdfda251fc96c91d7972
Oct 30 19:35:01.464427 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.464391    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:01.464511 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.464460    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:01.464554 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.464491    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:01.707165 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.707126    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:01.902080 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.902043    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57" event={"ID":"bcdfb861-99cb-4c80-882f-fed7871177c2","Type":"ContainerStarted","Data":"88a50754e8ee0324f4dd9c846ace2d19f8569e0a86bc1e7ddf44ab302c8abb8f"}
Oct 30 19:35:01.902280 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.902103    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57" event={"ID":"bcdfb861-99cb-4c80-882f-fed7871177c2","Type":"ContainerStarted","Data":"7d3a9b178997a540c8a0f786973b3c8705f20befd960de09f5c89d284e05d31b"}
Oct 30 19:35:01.903801 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.903770    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25" event={"ID":"e42358a7-0f5b-476a-831a-cefabde3984e","Type":"ContainerStarted","Data":"5bb9d130295a6d0be7045bd5a9f972f411b9201c510849a6a868f634152476c2"}
Oct 30 19:35:01.903931 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.903820    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25" event={"ID":"e42358a7-0f5b-476a-831a-cefabde3984e","Type":"ContainerStarted","Data":"fb4a421533d0a87e448c660530238ab1b0439603f8d5fdfda251fc96c91d7972"}
Oct 30 19:35:01.938396 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.938346    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:01.962979 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:01.962866    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57" podStartSLOduration=1.9628521110000001 podStartE2EDuration="1.962852111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:01.960866142 +0000 UTC m=+855.260128271" watchObservedRunningTime="2026-10-30 19:35:01.962852111 +0000 UTC m=+855.262114239"
Oct 30 19:35:02.013485 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:02.013407    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25" podStartSLOduration=2.013387318 podStartE2EDuration="2.013387318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:02.010038365 +0000 UTC m=+855.309300495" watchObservedRunningTime="2026-10-30 19:35:02.013387318 +0000 UTC m=+855.312649454"
Oct 30 19:35:02.160670 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:02.160628    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:02.162602 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:02.162576    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:35:02.162796 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:02.162775    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25" podUID="e42358a7-0f5b-476a-831a-cefabde3984e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:35:02.959337 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:02.958505    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:35:02.959337 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:02.958559    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57" podUID="bcdfb861-99cb-4c80-882f-fed7871177c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 19:35:03.161503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:03.161468    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:35:03.161748 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:03.161518    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25" podUID="e42358a7-0f5b-476a-831a-cefabde3984e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:35:03.528388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:03.527960    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:03.939793 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:03.939759    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:35:03.940096 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:03.940053    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57" podUID="bcdfb861-99cb-4c80-882f-fed7871177c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 19:35:04.072857 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:04.072822    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:35:04.085769 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:04.085742    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:35:04.160753 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:04.160715    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:35:04.160910 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:04.160769    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25" podUID="e42358a7-0f5b-476a-831a-cefabde3984e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:35:04.940390 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:04.939645    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:35:04.940390 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:04.939712    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57" podUID="bcdfb861-99cb-4c80-882f-fed7871177c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 19:35:04.981830 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:04.981798    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:35:05.160726 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:05.160695    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:35:05.161197 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:05.161157    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25" podUID="e42358a7-0f5b-476a-831a-cefabde3984e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:35:05.186504 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:05.186472    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:05.939385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:05.938718    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:35:05.939385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:05.938777    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57" podUID="bcdfb861-99cb-4c80-882f-fed7871177c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 19:35:06.161398 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:06.161350    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:35:06.161783 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:06.161417    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25" podUID="e42358a7-0f5b-476a-831a-cefabde3984e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:35:06.382940 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:35:06.382908    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:35:06.938808 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:06.938766    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:35:06.948386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:06.939069    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57" podUID="bcdfb861-99cb-4c80-882f-fed7871177c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 19:35:07.103028 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:07.102993    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:07.173741 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:07.172889    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:35:07.173741 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:07.172949    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25" podUID="e42358a7-0f5b-476a-831a-cefabde3984e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:35:07.939149 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:07.939114    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:35:07.939325 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:07.939185    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57" podUID="bcdfb861-99cb-4c80-882f-fed7871177c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 19:35:08.161413 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:08.161376    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:35:08.161607 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:08.161441    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25" podUID="e42358a7-0f5b-476a-831a-cefabde3984e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:35:08.749991 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:08.749949    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:08.942654 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:08.942626    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:08.942999 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:08.942979    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:08.943694 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:08.943679    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:35:09.165449 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:09.165420    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:09.165830 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:09.165801    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:09.166375 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:09.166343    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:35:10.411804 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:10.411767    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:12.587385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:12.583762    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:35:12.605470 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:12.605424    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:35:12.660214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:12.660174    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:35:12.661311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:12.661281    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:35:12.810452 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:12.803281    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:35:12.825556 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:12.825520    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:35:12.878679 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:12.878639    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:35:13.545565 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:13.545534    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:14.880170 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:14.880132    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"]
Oct 30 19:35:14.886030 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:14.886005    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:14.924735 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:14.924698    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"]
Oct 30 19:35:14.950250 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:14.950217    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"]
Oct 30 19:35:14.954825 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:14.954801    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:14.970466 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:14.970440    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"]
Oct 30 19:35:14.972021 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:14.971964    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcrx5\" (UniqueName: \"kubernetes.io/projected/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-kube-api-access-gcrx5\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:14.972507 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:14.972343    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-podinfo\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:14.972507 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:14.972436    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-workload-certs\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:14.972507 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:14.972473    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-workload-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:14.972507 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:14.972497    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-token\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:14.972834 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:14.972587    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-credential-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:14.972834 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:14.972654    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-data\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:14.972834 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:14.972690    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-envoy\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:14.972834 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:14.972709    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istiod-ca-cert\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.073533 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.073502    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-podinfo\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.073674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.073575    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-workload-certs\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.073674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.073606    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-workload-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.073674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.073628    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-token\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.073674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.073655    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-credential-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.073924 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.073703    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-workload-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.073924 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.073747    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istiod-ca-cert\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.073924 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.073785    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-workload-certs\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.073924 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.073818    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-envoy\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.073924 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.073844    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-podinfo\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.073924 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.073883    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-credential-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.074226 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.073953    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-data\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.074226 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.073988    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-envoy\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.074226 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.074017    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-data\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.074226 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.074033    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-workload-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.074226 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.074044    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5scbd\" (UniqueName: \"kubernetes.io/projected/b8c1988d-3a60-4e33-b9f2-c9878beb006b-kube-api-access-5scbd\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.074582 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.074283    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-data\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.074876 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.074852    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-credential-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.074963 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.074945    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istiod-ca-cert\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.075581 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.075555    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istiod-ca-cert\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.075883 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.075853    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcrx5\" (UniqueName: \"kubernetes.io/projected/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-kube-api-access-gcrx5\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.075954 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.075907    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-token\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.076594 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.076558    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-workload-certs\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.076803 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.076780    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-podinfo\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.079838 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.079810    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-envoy\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.096053 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.096026    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcrx5\" (UniqueName: \"kubernetes.io/projected/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-kube-api-access-gcrx5\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.096624 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.096599    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-token\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.177263 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.177203    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-data\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.177263 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.177235    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5scbd\" (UniqueName: \"kubernetes.io/projected/b8c1988d-3a60-4e33-b9f2-c9878beb006b-kube-api-access-5scbd\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.177460 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.177310    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-token\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.177460 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.177374    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-credential-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.177460 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.177409    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-workload-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.177620 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.177533    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istiod-ca-cert\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.177620 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.177577    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-data\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.177710 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.177580    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-workload-certs\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.177710 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.177653    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-envoy\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.177710 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.177660    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-workload-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.177710 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.177680    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-podinfo\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.177882 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.177719    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-credential-socket\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.177882 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.177846    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-workload-certs\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.178211 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.178193    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istiod-ca-cert\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.179659 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.179638    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-envoy\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.180021 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.180004    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-podinfo\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.196878 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.196858    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5scbd\" (UniqueName: \"kubernetes.io/projected/b8c1988d-3a60-4e33-b9f2-c9878beb006b-kube-api-access-5scbd\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.197016 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.196935    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-token\") pod \"e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") " pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.210929 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.210902    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:15.220770 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.220741    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:15.278116 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.277845    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:15.534623 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:35:15.534595    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c1988d_3a60_4e33_b9f2_c9878beb006b.slice/crio-9302bb8386063a07ada4392a720274e3677537e361e31f7798e2e3854c262be2 WatchSource:0}: Error finding container 9302bb8386063a07ada4392a720274e3677537e361e31f7798e2e3854c262be2: Status 404 returned error can't find the container with id 9302bb8386063a07ada4392a720274e3677537e361e31f7798e2e3854c262be2
Oct 30 19:35:15.534931 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.534910    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"]
Oct 30 19:35:15.536928 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.536894    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:15.537018 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.536953    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:15.537018 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.536985    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:15.598925 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:35:15.598900    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a18b97_a6fc_4f33_848b_86bb8ef66fc1.slice/crio-ac10d95d5debaad2606c26fba81f3290507eaafa97ff076d31e6e3e6f3432b42 WatchSource:0}: Error finding container ac10d95d5debaad2606c26fba81f3290507eaafa97ff076d31e6e3e6f3432b42: Status 404 returned error can't find the container with id ac10d95d5debaad2606c26fba81f3290507eaafa97ff076d31e6e3e6f3432b42
Oct 30 19:35:15.599652 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.599630    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"]
Oct 30 19:35:15.604556 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.602193    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:15.604556 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.602339    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:15.604556 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:15.602524    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:16.089545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:16.089506    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5" event={"ID":"b8c1988d-3a60-4e33-b9f2-c9878beb006b","Type":"ContainerStarted","Data":"43c6b1e76bc887c479abb46e7f5409e163543947cf7f170e262164ae676ada3d"}
Oct 30 19:35:16.090035 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:16.089607    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5" event={"ID":"b8c1988d-3a60-4e33-b9f2-c9878beb006b","Type":"ContainerStarted","Data":"9302bb8386063a07ada4392a720274e3677537e361e31f7798e2e3854c262be2"}
Oct 30 19:35:16.091303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:16.091270    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5" event={"ID":"82a18b97-a6fc-4f33-848b-86bb8ef66fc1","Type":"ContainerStarted","Data":"b14f96505bb2a9ae7d78b914c27b646e6e155b32e9fa679b7d9de0bd0cf62753"}
Oct 30 19:35:16.091446 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:16.091328    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5" event={"ID":"82a18b97-a6fc-4f33-848b-86bb8ef66fc1","Type":"ContainerStarted","Data":"ac10d95d5debaad2606c26fba81f3290507eaafa97ff076d31e6e3e6f3432b42"}
Oct 30 19:35:16.138841 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:16.138776    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5" podStartSLOduration=2.138755691 podStartE2EDuration="2.138755691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:16.134841339 +0000 UTC m=+869.434103469" watchObservedRunningTime="2026-10-30 19:35:16.138755691 +0000 UTC m=+869.438017820"
Oct 30 19:35:16.182935 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:16.182875    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5" podStartSLOduration=2.182860347 podStartE2EDuration="2.182860347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:16.180165258 +0000 UTC m=+869.479427389" watchObservedRunningTime="2026-10-30 19:35:16.182860347 +0000 UTC m=+869.482122537"
Oct 30 19:35:16.220396 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:16.214541    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:16.220396 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:16.214722    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:35:16.220396 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:16.214767    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5" podUID="b8c1988d-3a60-4e33-b9f2-c9878beb006b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:35:16.278716 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:16.278683    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:16.280185 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:16.280162    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:35:16.280307 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:16.280201    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5" podUID="82a18b97-a6fc-4f33-848b-86bb8ef66fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:35:17.095084 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:17.095046    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:17.230386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:17.229764    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:35:17.230386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:17.229831    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5" podUID="b8c1988d-3a60-4e33-b9f2-c9878beb006b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:35:17.279523 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:17.279165    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:35:17.279523 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:17.279211    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5" podUID="82a18b97-a6fc-4f33-848b-86bb8ef66fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:35:18.214704 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:18.214670    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:35:18.215292 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:18.215260    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5" podUID="b8c1988d-3a60-4e33-b9f2-c9878beb006b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:35:18.279110 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:18.279076    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:35:18.279257 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:18.279114    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5" podUID="82a18b97-a6fc-4f33-848b-86bb8ef66fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:35:19.232385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:19.214675    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:35:19.232385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:19.214745    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5" podUID="b8c1988d-3a60-4e33-b9f2-c9878beb006b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:35:19.278755 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:19.278723    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:35:19.278915 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:19.278788    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5" podUID="82a18b97-a6fc-4f33-848b-86bb8ef66fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:35:20.219846 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:20.213693    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:35:20.219846 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:20.213754    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5" podUID="b8c1988d-3a60-4e33-b9f2-c9878beb006b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:35:20.279666 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:20.279627    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:35:20.280068 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:20.279690    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5" podUID="82a18b97-a6fc-4f33-848b-86bb8ef66fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:35:20.359924 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:20.359896    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:21.231386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:21.212761    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:35:21.231386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:21.212832    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5" podUID="b8c1988d-3a60-4e33-b9f2-c9878beb006b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:35:21.279488 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:21.279453    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:35:21.279660 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:21.279522    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5" podUID="82a18b97-a6fc-4f33-848b-86bb8ef66fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:35:22.157218 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.157186    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"]
Oct 30 19:35:22.161565 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.161539    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.183934 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.183903    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"]
Oct 30 19:35:22.189863 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.189835    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d797d2\"/\"distinct-2-w1-d797d2-kserve-self-signed-certs\""
Oct 30 19:35:22.207092 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.207060    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-dshm\") pod \"distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.207319 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.207296    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e640ac87-e1bd-4f61-8848-f82dd4835442-tls-certs\") pod \"distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.207593 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.207567    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-model-cache\") pod \"distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.207702 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.207687    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-tmp-dir\") pod \"distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.207757 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.207730    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-274f4\" (UniqueName: \"kubernetes.io/projected/e640ac87-e1bd-4f61-8848-f82dd4835442-kube-api-access-274f4\") pod \"distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.207816 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.207796    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-home\") pod \"distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.211969 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.211943    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:35:22.212054 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.211988    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5" podUID="b8c1988d-3a60-4e33-b9f2-c9878beb006b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:35:22.280546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.280515    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:35:22.280683 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.280571    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5" podUID="82a18b97-a6fc-4f33-848b-86bb8ef66fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:35:22.290647 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.290617    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:22.309426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.309389    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-model-cache\") pod \"distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.309575 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.309512    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-tmp-dir\") pod \"distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.309575 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.309555    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-274f4\" (UniqueName: \"kubernetes.io/projected/e640ac87-e1bd-4f61-8848-f82dd4835442-kube-api-access-274f4\") pod \"distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.309748 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.309621    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-home\") pod \"distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.309748 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.309652    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-dshm\") pod \"distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.309748 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.309718    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e640ac87-e1bd-4f61-8848-f82dd4835442-tls-certs\") pod \"distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.309980 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.309757    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-model-cache\") pod \"distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.309980 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.309915    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-tmp-dir\") pod \"distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.310182 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.310154    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-home\") pod \"distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.312040 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.312018    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-dshm\") pod \"distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.313052 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.313032    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e640ac87-e1bd-4f61-8848-f82dd4835442-tls-certs\") pod \"distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.332481 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.332452    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-274f4\" (UniqueName: \"kubernetes.io/projected/e640ac87-e1bd-4f61-8848-f82dd4835442-kube-api-access-274f4\") pod \"distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") " pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.492898 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.492593    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:22.936993 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:22.936956    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"]
Oct 30 19:35:22.938005 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:35:22.937979    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode640ac87_e1bd_4f61_8848_f82dd4835442.slice/crio-42c6e57cf0d1a007ba3725270f7d17eddcf4550f138ff1056ec3692aed4ba791 WatchSource:0}: Error finding container 42c6e57cf0d1a007ba3725270f7d17eddcf4550f138ff1056ec3692aed4ba791: Status 404 returned error can't find the container with id 42c6e57cf0d1a007ba3725270f7d17eddcf4550f138ff1056ec3692aed4ba791
Oct 30 19:35:23.200448 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:23.200334    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" event={"ID":"e640ac87-e1bd-4f61-8848-f82dd4835442","Type":"ContainerStarted","Data":"5bf841fa8a53fbc255a62c00e33fedd2bc42b7e233c7757eba6b88a0777d61c9"}
Oct 30 19:35:23.200448 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:23.200395    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" event={"ID":"e640ac87-e1bd-4f61-8848-f82dd4835442","Type":"ContainerStarted","Data":"42c6e57cf0d1a007ba3725270f7d17eddcf4550f138ff1056ec3692aed4ba791"}
Oct 30 19:35:23.215558 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:23.215532    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:23.215823 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:23.215803    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:23.216445 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:23.216429    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:35:23.238322 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:23.238290    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:23.242922 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:23.242883    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podStartSLOduration=1.242871767 podStartE2EDuration="1.242871767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:23.241520093 +0000 UTC m=+876.540782222" watchObservedRunningTime="2026-10-30 19:35:23.242871767 +0000 UTC m=+876.542133895"
Oct 30 19:35:23.284560 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:23.284526    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:23.284961 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:23.284940    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:23.285816 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:23.285787    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:35:23.493883 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:23.493790    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:25.468505 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:25.468477    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:27.202218 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:27.202177    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:28.802708 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:28.802671    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:30.459969 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.459931    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"]
Oct 30 19:35:30.464092 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.464070    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.513822 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.513755    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-token\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.514292 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.514240    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-workload-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.514444 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.514312    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-envoy\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.514497 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.514445    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-data\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.515685 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.514555    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-credential-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.515685 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.514673    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-podinfo\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.515685 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.514767    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb94f9e-d211-4973-86f8-72b96cf14270-istiod-ca-cert\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.515685 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.514792    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk7bw\" (UniqueName: \"kubernetes.io/projected/1eb94f9e-d211-4973-86f8-72b96cf14270-kube-api-access-gk7bw\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.515685 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.514846    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-workload-certs\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.516009 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.515841    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"]
Oct 30 19:35:30.523642 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.523616    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"]
Oct 30 19:35:30.524819 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.524790    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:30.528734 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.528716    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.537272 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.537246    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"]
Oct 30 19:35:30.616340 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.616312    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-credential-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.616565 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.616345    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-workload-certs\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.616565 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.616383    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-token\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.616565 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.616525    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-podinfo\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.616727 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.616586    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-podinfo\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.616727 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.616638    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb94f9e-d211-4973-86f8-72b96cf14270-istiod-ca-cert\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.616727 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.616664    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk7bw\" (UniqueName: \"kubernetes.io/projected/1eb94f9e-d211-4973-86f8-72b96cf14270-kube-api-access-gk7bw\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.616949 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.616759    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-workload-certs\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.616949 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.616816    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-credential-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.616949 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.616835    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-token\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.616949 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.616884    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istiod-ca-cert\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.616949 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.616939    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-workload-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.617189 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.617059    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-envoy\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.617189 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.617095    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-workload-certs\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.617189 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.617132    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-workload-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.617189 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.617161    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-credential-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.617376 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.617216    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-envoy\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.617376 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.617295    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-data\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.617376 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.617325    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-data\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.617376 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.617354    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4xvb\" (UniqueName: \"kubernetes.io/projected/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-kube-api-access-z4xvb\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.617593 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.617501    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-workload-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.617593 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.617539    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb94f9e-d211-4973-86f8-72b96cf14270-istiod-ca-cert\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.617674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.617646    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-data\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.619281 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.619259    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-podinfo\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.619386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.619328    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-envoy\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.636478 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.636453    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-token\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.636586 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.636571    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk7bw\" (UniqueName: \"kubernetes.io/projected/1eb94f9e-d211-4973-86f8-72b96cf14270-kube-api-access-gk7bw\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.718962 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.718895    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-workload-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.718962 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.718938    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-envoy\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.719157 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.718971    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-credential-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.719157 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.719014    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-data\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.719157 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.719033    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4xvb\" (UniqueName: \"kubernetes.io/projected/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-kube-api-access-z4xvb\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.719157 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.719095    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-workload-certs\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.719157 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.719116    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-token\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.719470 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.719219    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-podinfo\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.719470 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.719269    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-workload-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.719470 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.719339    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istiod-ca-cert\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.719470 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.719419    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-credential-socket\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.719676 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.719472    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-data\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.719732 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.719714    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-workload-certs\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.719945 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.719925    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istiod-ca-cert\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.721172 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.721150    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-envoy\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.721504 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.721488    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-podinfo\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.739045 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.739020    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-token\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.739527 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.739507    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4xvb\" (UniqueName: \"kubernetes.io/projected/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-kube-api-access-z4xvb\") pod \"e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") " pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:30.779707 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.779684    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:30.848595 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:30.848565    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:31.092525 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:31.092493    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"]
Oct 30 19:35:31.092675 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:35:31.092573    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb94f9e_d211_4973_86f8_72b96cf14270.slice/crio-d97fb25084d9eeede1d060ec187e747f53d1d2fd40507932ebaaa7b859b7798d WatchSource:0}: Error finding container d97fb25084d9eeede1d060ec187e747f53d1d2fd40507932ebaaa7b859b7798d: Status 404 returned error can't find the container with id d97fb25084d9eeede1d060ec187e747f53d1d2fd40507932ebaaa7b859b7798d
Oct 30 19:35:31.095047 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:31.095019    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:31.095129 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:31.095081    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:31.095129 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:31.095108    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:31.165992 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:35:31.165956    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91c1dde_2ad0_4d1e_9846_6d7db14fc6f2.slice/crio-fb1b5dc04afdf608c4f49587a67d46df3a9daad1edc1342389a53cc1c4238cd8 WatchSource:0}: Error finding container fb1b5dc04afdf608c4f49587a67d46df3a9daad1edc1342389a53cc1c4238cd8: Status 404 returned error can't find the container with id fb1b5dc04afdf608c4f49587a67d46df3a9daad1edc1342389a53cc1c4238cd8
Oct 30 19:35:31.167025 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:31.167001    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"]
Oct 30 19:35:31.168439 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:31.168408    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:31.168535 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:31.168479    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:31.168535 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:31.168524    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:31.217887 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:31.217864    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:31.293000 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:31.292966    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7" event={"ID":"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2","Type":"ContainerStarted","Data":"fb1b5dc04afdf608c4f49587a67d46df3a9daad1edc1342389a53cc1c4238cd8"}
Oct 30 19:35:31.315076 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:31.315032    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5" event={"ID":"1eb94f9e-d211-4973-86f8-72b96cf14270","Type":"ContainerStarted","Data":"8f2cb52aade6b04507d0118bf08dc7e14676be7726ecab85b597f50b926a3e86"}
Oct 30 19:35:31.315354 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:31.315330    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5" event={"ID":"1eb94f9e-d211-4973-86f8-72b96cf14270","Type":"ContainerStarted","Data":"d97fb25084d9eeede1d060ec187e747f53d1d2fd40507932ebaaa7b859b7798d"}
Oct 30 19:35:31.791969 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:31.791712    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:32.396205 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:32.396169    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:32.401412 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:32.400810    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7" event={"ID":"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2","Type":"ContainerStarted","Data":"e137656526b4b1397d4622cfa3721fcda8b8f39954088789b2796653f5cf1434"}
Oct 30 19:35:32.495386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:32.484236    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5" podStartSLOduration=2.484216173 podStartE2EDuration="2.484216173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:31.378832421 +0000 UTC m=+884.678094554" watchObservedRunningTime="2026-10-30 19:35:32.484216173 +0000 UTC m=+885.783478303"
Oct 30 19:35:32.496977 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:32.496907    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7" podStartSLOduration=2.496888251 podStartE2EDuration="2.496888251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:32.481659482 +0000 UTC m=+885.780921612" watchObservedRunningTime="2026-10-30 19:35:32.496888251 +0000 UTC m=+885.796150395"
Oct 30 19:35:32.505104 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:32.505075    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:32.787623 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:32.783714    4712 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:35:32.787623 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:32.783767    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5" podUID="1eb94f9e-d211-4973-86f8-72b96cf14270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 19:35:32.852265 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:32.852230    4712 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:35:32.872311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:32.872274    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7" podUID="d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 19:35:32.946385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:32.942795    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:32.946385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:32.942913    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:33.472291 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:33.472260    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:35:33.781166 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:33.781079    4712 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:35:33.781166 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:33.781140    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5" podUID="1eb94f9e-d211-4973-86f8-72b96cf14270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 19:35:33.857759 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:33.857713    4712 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:35:33.858344 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:33.857780    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7" podUID="d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 19:35:34.239851 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:34.233952    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:34.780433 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:34.780399    4712 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:35:34.780613 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:34.780472    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5" podUID="1eb94f9e-d211-4973-86f8-72b96cf14270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 19:35:34.855383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:34.854917    4712 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:35:34.855383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:34.854971    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7" podUID="d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 19:35:35.781495 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:35.781192    4712 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:35:35.782877 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:35.782707    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5" podUID="1eb94f9e-d211-4973-86f8-72b96cf14270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 19:35:35.790278 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:35.789345    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:35.853101 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:35.853064    4712 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:35:35.853355 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:35.853327    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7" podUID="d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 19:35:36.384715 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:35:36.384681    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:35:36.781389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:36.781297    4712 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:35:36.781389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:36.781346    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5" podUID="1eb94f9e-d211-4973-86f8-72b96cf14270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 19:35:36.857700 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:36.857667    4712 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:35:36.858196 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:36.857971    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7" podUID="d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 19:35:37.428671 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:37.428632    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:37.627527 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:37.627489    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:37.784308 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:37.783753    4712 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:35:37.784308 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:37.783832    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5" podUID="1eb94f9e-d211-4973-86f8-72b96cf14270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 19:35:37.849324 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:37.849282    4712 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:35:37.849537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:37.849340    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7" podUID="d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 19:35:38.974906 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:38.974876    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:38.975806 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:38.975780    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:38.976822 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:38.976800    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:35:39.009443 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:39.009405    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:39.010327 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:39.010304    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:39.011427 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:39.011352    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:35:39.033454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:39.033423    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:40.688441 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:40.688407    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:41.897004 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:41.896972    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:42.033120 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:42.033091    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:42.258722 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:42.258606    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:43.765873 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:43.765827    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:45.302440 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:45.302402    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:45.995998 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:45.995966    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"]
Oct 30 19:35:46.000217 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.000196    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.032117 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.032085    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"]
Oct 30 19:35:46.069139 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.069106    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-workload-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.069271 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.069172    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-credential-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.069317 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.069274    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istiod-ca-cert\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.069353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.069323    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-podinfo\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.069474 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.069454    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr4ss\" (UniqueName: \"kubernetes.io/projected/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-kube-api-access-dr4ss\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.069511 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.069485    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-workload-certs\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.069547 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.069514    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-token\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.069626 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.069608    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-data\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.069713 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.069697    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-envoy\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.171207 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.171180    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-workload-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.171326 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.171223    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-credential-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.171326 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.171259    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istiod-ca-cert\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.171326 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.171295    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-podinfo\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.171459 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.171408    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr4ss\" (UniqueName: \"kubernetes.io/projected/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-kube-api-access-dr4ss\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.171459 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.171434    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-workload-certs\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.171545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.171473    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-token\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.171545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.171512    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-data\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.171649 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.171592    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-envoy\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.171649 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.171632    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-credential-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.171750 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.171688    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-workload-socket\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.171899 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.171876    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-workload-certs\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.171955 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.171883    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-data\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.172263 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.172246    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istiod-ca-cert\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.173704 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.173686    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-envoy\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.173927 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.173908    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-podinfo\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.192081 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.192054    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-token\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.192254 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.192233    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr4ss\" (UniqueName: \"kubernetes.io/projected/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-kube-api-access-dr4ss\") pod \"e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") " pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.314098 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.314033    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:46.624946 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:35:46.624902    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0a8af7_9b2d_4a53_8be6_703ed9ab2755.slice/crio-9434c95f66e3a01798f4101a24f7e726908e4f47a1e3d83c1a0ebddafbdd6460 WatchSource:0}: Error finding container 9434c95f66e3a01798f4101a24f7e726908e4f47a1e3d83c1a0ebddafbdd6460: Status 404 returned error can't find the container with id 9434c95f66e3a01798f4101a24f7e726908e4f47a1e3d83c1a0ebddafbdd6460
Oct 30 19:35:46.625301 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.625278    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"]
Oct 30 19:35:46.627463 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.627429    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:46.627537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.627493    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:46.627537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.627525    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:35:46.958887 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.958799    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:35:46.961411 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:46.961356    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:35:47.060408 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:47.060344    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:47.548524 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:47.548481    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl" event={"ID":"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755","Type":"ContainerStarted","Data":"e8fc7148ef5730665cb0c40f8f37b4cc0aa2a8c2f3dc39a888655b089204ed0d"}
Oct 30 19:35:47.548978 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:47.548531    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl" event={"ID":"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755","Type":"ContainerStarted","Data":"9434c95f66e3a01798f4101a24f7e726908e4f47a1e3d83c1a0ebddafbdd6460"}
Oct 30 19:35:47.603692 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:47.603646    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl" podStartSLOduration=2.603633596 podStartE2EDuration="2.603633596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:47.60156165 +0000 UTC m=+900.900823778" watchObservedRunningTime="2026-10-30 19:35:47.603633596 +0000 UTC m=+900.902895725"
Oct 30 19:35:48.325679 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:48.325646    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:48.329634 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:48.329608    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:35:48.329769 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:48.329679    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl" podUID="9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:35:48.440898 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:48.440836    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:49.321460 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:49.321422    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:35:49.322067 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:49.322035    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl" podUID="9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:35:50.239385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:50.228948    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:50.316743 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:50.316705    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:35:50.317023 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:50.316994    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl" podUID="9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:35:51.320750 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:51.320712    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:35:51.321395 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:51.321336    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl" podUID="9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:35:51.832053 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:51.832024    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:52.324387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:52.317693    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:35:52.324387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:52.317763    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl" podUID="9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:35:53.197165 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.197129    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"]
Oct 30 19:35:53.203577 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.203552    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.219280 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.219255    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"]
Oct 30 19:35:53.226504 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.226468    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-544fc8\"/\"distinct-2-w5-544fc8-kserve-self-signed-certs\""
Oct 30 19:35:53.290636 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.290608    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:53.293328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.293291    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/946e2596-1023-4d05-ba84-2dd3da74cc8e-tls-certs\") pod \"distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.293481 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.293378    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-tmp-dir\") pod \"distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.293481 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.293437    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-model-cache\") pod \"distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.293481 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.293457    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slrzx\" (UniqueName: \"kubernetes.io/projected/946e2596-1023-4d05-ba84-2dd3da74cc8e-kube-api-access-slrzx\") pod \"distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.293667 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.293589    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-home\") pod \"distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.293667 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.293625    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-dshm\") pod \"distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.315343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.315309    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:35:53.315457 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.315434    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl" podUID="9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:35:53.395008 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.394978    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/946e2596-1023-4d05-ba84-2dd3da74cc8e-tls-certs\") pod \"distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.395890 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.395867    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-tmp-dir\") pod \"distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.396324 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.396301    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-tmp-dir\") pod \"distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.396503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.396418    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-model-cache\") pod \"distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.396666 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.396645    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slrzx\" (UniqueName: \"kubernetes.io/projected/946e2596-1023-4d05-ba84-2dd3da74cc8e-kube-api-access-slrzx\") pod \"distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.397716 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.397695    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-home\") pod \"distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.398061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.398038    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-home\") pod \"distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.408321 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.397795    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-model-cache\") pod \"distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.408321 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.407428    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-dshm\") pod \"distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.421396 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.418959    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/946e2596-1023-4d05-ba84-2dd3da74cc8e-tls-certs\") pod \"distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.428763 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.428734    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-dshm\") pod \"distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.430738 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.430717    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slrzx\" (UniqueName: \"kubernetes.io/projected/946e2596-1023-4d05-ba84-2dd3da74cc8e-kube-api-access-slrzx\") pod \"distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") " pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.524669 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.524574    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:53.530524 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.530496    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:53.874335 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:53.874301    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"]
Oct 30 19:35:54.318392 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:54.318306    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:54.318634 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:54.318610    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:54.319260 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:54.319241    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:35:54.597807 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:54.597711    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" event={"ID":"946e2596-1023-4d05-ba84-2dd3da74cc8e","Type":"ContainerStarted","Data":"350b4d281e03a0648a9702028d22d689f3fcd16b4f332e4d6eb15c207f9f61de"}
Oct 30 19:35:54.597807 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:54.597753    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" event={"ID":"946e2596-1023-4d05-ba84-2dd3da74cc8e","Type":"ContainerStarted","Data":"ad017e08bd3bb3069aed4482b2b2ab8fa43948cd512e32a9e8cbaae66e19a0e3"}
Oct 30 19:35:54.636399 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:54.636339    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podStartSLOduration=1.636326794 podStartE2EDuration="1.636326794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:54.635021903 +0000 UTC m=+907.934284032" watchObservedRunningTime="2026-10-30 19:35:54.636326794 +0000 UTC m=+907.935588978"
Oct 30 19:35:55.128872 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:55.128835    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:55.525475 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:55.525397    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:35:56.653737 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:56.653700    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:58.183741 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:58.183702    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:35:59.703553 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:35:59.703516    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:00.319124 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.319061    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"]
Oct 30 19:36:00.333849 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.333819    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.348111 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.348075    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"]
Oct 30 19:36:00.517388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.513307    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.540385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.517958    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.540710 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.540681    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsbg2\" (UniqueName: \"kubernetes.io/projected/7f364e92-b446-4671-8a42-573ebe98d729-kube-api-access-xsbg2\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.540914 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.540894    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f364e92-b446-4671-8a42-573ebe98d729-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.541164 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.541146    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.541596 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.541574    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f364e92-b446-4671-8a42-573ebe98d729-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.562393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.541936    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f364e92-b446-4671-8a42-573ebe98d729-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.562766 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.562740    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.563004 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.562984    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.698008 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.697970    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f364e92-b446-4671-8a42-573ebe98d729-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.732526 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.732483    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.733163 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.733136    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.760385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.736548    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.760685 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.760655    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.764381 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.760825    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsbg2\" (UniqueName: \"kubernetes.io/projected/7f364e92-b446-4671-8a42-573ebe98d729-kube-api-access-xsbg2\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.772383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.768428    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f364e92-b446-4671-8a42-573ebe98d729-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.776469 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.776442    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.785491 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.785465    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f364e92-b446-4671-8a42-573ebe98d729-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.834385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.800316    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.834385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.712518    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f364e92-b446-4671-8a42-573ebe98d729-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.834385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.815238    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.858124 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.858090    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.858743 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.858715    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.862391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.860423    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.887988 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.887955    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f364e92-b446-4671-8a42-573ebe98d729-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.942386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.942188    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsbg2\" (UniqueName: \"kubernetes.io/projected/7f364e92-b446-4671-8a42-573ebe98d729-kube-api-access-xsbg2\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.961502 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.952497    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f364e92-b446-4671-8a42-573ebe98d729-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:00.993389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:00.991997    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:01.493393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:01.492455    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:01.518485 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:01.518444    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:01.582403 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:01.581793    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:36:01.632508 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:01.632447    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:01.659468 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:01.658446    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:01.660113 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:01.659762    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:36:01.719386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:01.718453    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:01.911537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:01.824490    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:01.911537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:01.834611    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:01.911537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:01.835284    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:01.911537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:01.867471    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:02.108169 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.096179    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:36:02.320614 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.295470    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:02.477949 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.469750    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"]
Oct 30 19:36:02.508830 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.506459    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:02.622008 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.561616    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:36:02.622008 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.621516    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:36:02.622008 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.621626    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:36:02.622008 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.621693    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:36:02.650685 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.650275    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:02.652621 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.652574    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:02.710223 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.653578    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:36:02.710223 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.709943    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:36:02.714094 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.665072    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:02.862384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.687323    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:36:02.862384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.687289    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:02.862384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.738208    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:02.862384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.840668    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:02.862384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.840747    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:02.862384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.840801    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:36:02.862384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.840916    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:36:02.862384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.842118    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:36:03.068387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:02.957024    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:03.139499 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:03.139452    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:03.209456 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:03.209415    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded"
Oct 30 19:36:03.375388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:03.339261    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:36:03.527445 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:03.511537    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:03.625528 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:03.625477    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:03.886387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:03.885984    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:03.897390 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:03.896558    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:36:03.909405 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:03.905056    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:03.909405 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:03.905658    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Oct 30 19:36:03.909405 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:03.906547    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:36:03.909872 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:03.909835    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:04.046696 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.037046    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:04.052930 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.052842    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:36:04.053866 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.053836    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 30 19:36:04.054554 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.054528    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:36:04.055312 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.055285    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:04.108385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.089028    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:04.380388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.329682    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:04.475740 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.475691    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:04.517414 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:36:04.517381    4712 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.672s"
Oct 30 19:36:04.517749 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.517728    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:36:04.517925 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.517910    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:36:04.518069 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.518055    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:36:04.518214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.518201    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:36:04.518404 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.518347    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:36:04.518580 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.518565    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:36:04.518692 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.518673    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v" event={"ID":"7f364e92-b446-4671-8a42-573ebe98d729","Type":"ContainerStarted","Data":"852a67726a16c00b5559b85612e55587435b2700b9d3187f09019b4dd496cce5"}
Oct 30 19:36:04.518841 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.518826    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:36:04.519331 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.519311    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:36:04.519581 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.519564    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:36:04.533217 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.533056    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:36:04.533217 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.533083    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:36:04.533217 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.533166    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:36:04.533217 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.533201    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:36:04.533452 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.533272    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:36:04.533452 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.533290    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:36:04.533452 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.533325    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:36:04.533452 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.533414    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:36:04.533452 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.533446    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:36:04.533616 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.533474    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:36:04.533616 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.533502    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:36:04.533616 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.533529    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:36:04.557170 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.555550    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:36:04.562464 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.562423    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:36:04.563905 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:04.563888    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:36:05.288266 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:05.288223    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v" event={"ID":"7f364e92-b446-4671-8a42-573ebe98d729","Type":"ContainerStarted","Data":"51408411c28e15e383f2c73f744a85f84b91c5d6ee2ddc58b38d259552566cee"}
Oct 30 19:36:05.432898 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:05.432849    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:05.437808 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:05.437747    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v" podStartSLOduration=5.437732108 podStartE2EDuration="5.437732108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:05.429640637 +0000 UTC m=+918.728902767" watchObservedRunningTime="2026-10-30 19:36:05.437732108 +0000 UTC m=+918.736994236"
Oct 30 19:36:05.999717 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:05.999687    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:06.000866 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:06.000836    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:36:06.001004 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:06.000883    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v" podUID="7f364e92-b446-4671-8a42-573ebe98d729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 19:36:06.374818 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:36:06.374778    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:36:06.999316 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:06.999280    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:36:06.999519 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:06.999334    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v" podUID="7f364e92-b446-4671-8a42-573ebe98d729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 19:36:07.241901 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:07.241864    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:07.999328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:07.999297    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:36:07.999733 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:07.999345    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v" podUID="7f364e92-b446-4671-8a42-573ebe98d729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 19:36:08.941682 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:08.941631    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:08.999641 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:08.999605    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:36:09.000056 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:08.999671    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v" podUID="7f364e92-b446-4671-8a42-573ebe98d729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 19:36:09.999357 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:09.999323    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:36:09.999554 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:09.999414    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v" podUID="7f364e92-b446-4671-8a42-573ebe98d729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 19:36:10.491941 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:10.491901    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:10.999277 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:10.999238    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:36:10.999474 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:10.999300    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v" podUID="7f364e92-b446-4671-8a42-573ebe98d729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 19:36:11.978490 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:11.978434    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:12.003783 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:12.003753    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:12.004252 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:12.004233    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:12.005023 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:12.005003    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:36:13.665691 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:13.665565    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:15.153469 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:15.153424    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:16.864887 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:16.864846    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:18.394912 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:18.394878    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:20.082946 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:20.082906    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:21.795048 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:21.794993    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:23.531603 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:23.531565    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:25.110307 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:25.110276    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:26.744849 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:26.744809    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:28.639554 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:28.639520    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:30.220533 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:30.220474    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:30.509439 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:30.508812    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:30.521389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:30.518054    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:36:30.521389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:30.519305    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:36:30.536947 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:30.536900    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:30.537798 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:30.537764    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:36:30.538133 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:30.538073    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:36:30.560384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:30.556685    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:30.576234 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:30.567419    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:30.599175 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:30.599147    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"]
Oct 30 19:36:30.820909 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:30.820820    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"]
Oct 30 19:36:30.845386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:30.830645    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:30.877394 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:30.855666    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"]
Oct 30 19:36:31.025146 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:30.908422    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:36:31.025146 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:30.910804    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istiod-ca-cert\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.025146 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.022860    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-token\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.025146 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.022955    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-credential-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.025146 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.023061    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-envoy\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.025146 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.023167    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-podinfo\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.025146 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.023228    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-workload-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.025146 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.023278    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fxq8\" (UniqueName: \"kubernetes.io/projected/bb85c12b-5f17-4e75-9a68-96e9f83389b1-kube-api-access-8fxq8\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.025146 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.023313    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-data\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.025146 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.023416    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-workload-certs\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.025146 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:30.920644    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:31.041564 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:30.957924    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:31.179862 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.147322    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istiod-ca-cert\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.180394 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.180154    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-token\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.180394 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.180256    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-credential-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.202116 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.180376    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-envoy\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.202116 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.201005    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-podinfo\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.202116 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.201076    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-workload-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.202116 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.201130    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fxq8\" (UniqueName: \"kubernetes.io/projected/bb85c12b-5f17-4e75-9a68-96e9f83389b1-kube-api-access-8fxq8\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.202116 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.201162    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-data\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.202116 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.201242    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-workload-certs\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.202116 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.201665    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-workload-certs\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.202116 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.182299    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istiod-ca-cert\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.210593 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.209843    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-workload-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.210593 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.182595    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-credential-socket\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.210593 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.210549    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-data\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.279758 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.279720    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-podinfo\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.297055 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.295906    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:31.318829 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.317974    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:31.360383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.336467    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:36:31.360383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.336534    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc" podUID="b727df7a-56c7-46fc-974a-38aebc2593c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:31.379391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.378574    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.431483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.424394    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-envoy\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.431483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.424446    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-workload-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.431483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.424535    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcrjx\" (UniqueName: \"kubernetes.io/projected/028719db-ba4c-4f77-8f0f-45ed5d94e76c-kube-api-access-bcrjx\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.431483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.424580    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-token\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.431483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.424616    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-credential-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.431483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.424666    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-data\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.431483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.424765    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-podinfo\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.431483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.424818    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istiod-ca-cert\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.431483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.425067    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-workload-certs\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.538426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.511482    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:31.538426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.518424    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:36:31.538426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.522428    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:31.538426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.527398    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-envoy\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.538426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.527446    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-workload-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.538426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.527503    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcrjx\" (UniqueName: \"kubernetes.io/projected/028719db-ba4c-4f77-8f0f-45ed5d94e76c-kube-api-access-bcrjx\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.538426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.527548    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-token\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.538426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.527582    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-credential-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.538426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.527616    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-data\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.538426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.527705    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-podinfo\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.538426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.527743    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istiod-ca-cert\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.538426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.527920    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-workload-certs\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.538426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.528468    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-workload-certs\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.538426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.529716    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-workload-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.538426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.530311    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istiod-ca-cert\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.538426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.530561    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-data\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.593115 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.549962    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:36:31.593115 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.550005    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"]
Oct 30 19:36:31.593115 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.550071    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:36:31.593115 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.550110    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:36:31.593115 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.567968    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-podinfo\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.622424 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.593562    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:36:31.622424 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.593927    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:36:31.622424 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.594294    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:36:31.622424 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.594352    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:36:31.622424 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.596790    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:31.622424 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.597063    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:36:31.718760 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.623039    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:36:31.718760 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.623112    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:36:31.718760 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.623735    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:31.718760 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.624096    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:36:31.756958 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.728193    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:36:31.756958 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.728244    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd" podUID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:31.788054 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.788023    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-credential-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.819453 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.788629    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-token\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.828529 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.791302    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:31.828776 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.812877    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-envoy\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.838440 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.823059    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fxq8\" (UniqueName: \"kubernetes.io/projected/bb85c12b-5f17-4e75-9a68-96e9f83389b1-kube-api-access-8fxq8\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.881417 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.865209    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-token\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:31.908569 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.907203    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcrjx\" (UniqueName: \"kubernetes.io/projected/028719db-ba4c-4f77-8f0f-45ed5d94e76c-kube-api-access-bcrjx\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:31.908569 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:31.907293    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:32.063520 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.063477    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:32.064039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.064011    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:36:32.158310 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.064584    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:36:32.158643 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.064725    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:32.159033 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.138856    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:32.160120 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.158630    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:36:32.160602 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.160584    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:36:32.465387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.445635    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:32.466441 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.466414    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:36:32.518892 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.518844    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:32.522242 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.522210    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-envoy\") pod \"e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") " pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:32.536415 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.522485    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:32.536894 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.522897    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:32.537671 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.523051    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:32.627781 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.627729    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:36:32.630389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.628317    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:36:32.661441 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.640674    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:32.702394 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.698872    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:36:32.702394 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.699579    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:36:32.827030 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:32.826951    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:36:33.008394 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.008320    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:36:33.008579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.008402    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:36:33.008579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.008435    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:36:33.008579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.008466    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:36:33.008579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.008508    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:36:33.008579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.008543    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:36:33.008579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.008572    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:36:33.008904 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.008607    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:36:33.008904 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.008638    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:36:33.008904 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.008671    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:36:33.030345 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.030319    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"]
Oct 30 19:36:33.041079 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.041046    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:36:33.041382 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.041347    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:36:33.041512 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.041499    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:36:33.285923 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:36:33.285884    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb85c12b_5f17_4e75_9a68_96e9f83389b1.slice/crio-52276e4974d8ae155c9d3a54da838e62660cc10793492bce5ce2f6381205c820 WatchSource:0}: Error finding container 52276e4974d8ae155c9d3a54da838e62660cc10793492bce5ce2f6381205c820: Status 404 returned error can't find the container with id 52276e4974d8ae155c9d3a54da838e62660cc10793492bce5ce2f6381205c820
Oct 30 19:36:33.289570 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.289502    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:36:33.289692 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.289604    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:36:33.289692 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.289651    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:36:33.297956 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.297930    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"]
Oct 30 19:36:33.691184 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:33.691143    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:34.049611 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:34.049512    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb" event={"ID":"028719db-ba4c-4f77-8f0f-45ed5d94e76c","Type":"ContainerStarted","Data":"f6a37b740c8ee2bfa207effcd90788d19ba5b42cf5ede883ee26d15aca9e2b72"}
Oct 30 19:36:34.049611 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:34.049571    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb" event={"ID":"028719db-ba4c-4f77-8f0f-45ed5d94e76c","Type":"ContainerStarted","Data":"99bfea1a71f48462fe00e6e2f7d3e772879697e2218c486e4a3fdd195107881a"}
Oct 30 19:36:34.054746 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:34.054712    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh" event={"ID":"bb85c12b-5f17-4e75-9a68-96e9f83389b1","Type":"ContainerStarted","Data":"6f7ca7d42ef85b6955f6d058d4e7552c2677e695b0b3d74b5f863db8a962b564"}
Oct 30 19:36:34.054886 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:34.054770    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh" event={"ID":"bb85c12b-5f17-4e75-9a68-96e9f83389b1","Type":"ContainerStarted","Data":"52276e4974d8ae155c9d3a54da838e62660cc10793492bce5ce2f6381205c820"}
Oct 30 19:36:34.091962 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:34.091904    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb" podStartSLOduration=4.091892062 podStartE2EDuration="4.091892062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:34.091591292 +0000 UTC m=+947.390853422" watchObservedRunningTime="2026-10-30 19:36:34.091892062 +0000 UTC m=+947.391154190"
Oct 30 19:36:34.139337 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:34.139298    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:34.141483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:34.141445    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:36:34.141633 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:34.141503    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb" podUID="028719db-ba4c-4f77-8f0f-45ed5d94e76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:36:34.147902 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:34.147848    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh" podStartSLOduration=4.147830486 podStartE2EDuration="4.147830486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:34.143633013 +0000 UTC m=+947.442895143" watchObservedRunningTime="2026-10-30 19:36:34.147830486 +0000 UTC m=+947.447092616"
Oct 30 19:36:34.530888 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:34.530853    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:34.531285 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:34.531259    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:36:34.531465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:34.531438    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh" podUID="bb85c12b-5f17-4e75-9a68-96e9f83389b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:36:35.139824 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:35.139781    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:36:35.140473 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:35.140439    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb" podUID="028719db-ba4c-4f77-8f0f-45ed5d94e76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:36:35.538428 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:35.526444    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:36:35.538428 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:35.526504    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh" podUID="bb85c12b-5f17-4e75-9a68-96e9f83389b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:36:36.142793 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:36.142757    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:36:36.143445 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:36.143411    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb" podUID="028719db-ba4c-4f77-8f0f-45ed5d94e76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:36:36.387782 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:36:36.387748    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:36:36.459820 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:36.459687    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:36.533388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:36.532464    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:36:36.533388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:36.532528    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh" podUID="bb85c12b-5f17-4e75-9a68-96e9f83389b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:36:37.141386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:37.141229    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:36:37.141386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:37.141286    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb" podUID="028719db-ba4c-4f77-8f0f-45ed5d94e76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:36:37.534443 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:37.532685    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:36:37.534443 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:37.532751    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh" podUID="bb85c12b-5f17-4e75-9a68-96e9f83389b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:36:38.160409 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:38.159815    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:36:38.160409 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:38.159880    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb" podUID="028719db-ba4c-4f77-8f0f-45ed5d94e76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:36:38.536899 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:38.524293    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:36:38.536899 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:38.524381    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh" podUID="bb85c12b-5f17-4e75-9a68-96e9f83389b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:36:38.682647 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:38.678934    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:39.154634 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:39.154597    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:36:39.155133 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:39.155096    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb" podUID="028719db-ba4c-4f77-8f0f-45ed5d94e76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:36:39.526540 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:39.526450    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:36:39.526786 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:39.526755    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh" podUID="bb85c12b-5f17-4e75-9a68-96e9f83389b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:36:40.140242 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:40.140209    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:36:40.140668 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:40.140269    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb" podUID="028719db-ba4c-4f77-8f0f-45ed5d94e76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:36:40.524187 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:40.524089    4712 patch_prober.go:29] interesting pod/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:36:40.524187 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:40.524166    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh" podUID="bb85c12b-5f17-4e75-9a68-96e9f83389b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:36:41.144063 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:41.144034    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:41.144578 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:41.144554    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:41.145407 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:41.145352    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:36:41.528236 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:41.528155    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:41.528911 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:41.528880    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:41.529705 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:41.529683    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:36:41.719990 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:41.719957    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:44.237692 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:44.237656    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:45.485225 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:45.485185    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:46.136454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:46.136417    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:47.861084 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:47.861046    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:49.603710 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:49.603672    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:50.021095 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.021062    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"]
Oct 30 19:36:50.026257 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.026237    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.041217 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.041196    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6vscc\""
Oct 30 19:36:50.043643 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.043626    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 19:36:50.058153 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.058127    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"]
Oct 30 19:36:50.141998 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.141970    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 19:36:50.142157 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.142013    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 19:36:50.146706 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.146674    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6785a121-64c7-4514-8e16-aeba1d89ae02-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.146831 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.146721    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.146831 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.146820    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.146954 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.146917    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.147018 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.146954    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d55z8\" (UniqueName: \"kubernetes.io/projected/6785a121-64c7-4514-8e16-aeba1d89ae02-kube-api-access-d55z8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.147080 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.147056    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.147135 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.147124    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.248467 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.248440    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.248608 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.248480    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6785a121-64c7-4514-8e16-aeba1d89ae02-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.248608 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.248516    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.248608 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.248563    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.248723 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.248633    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.248723 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.248678    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d55z8\" (UniqueName: \"kubernetes.io/projected/6785a121-64c7-4514-8e16-aeba1d89ae02-kube-api-access-d55z8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.248807 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.248765    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.249444 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.249421    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.250976 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.250949    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.251094 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.250956    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6785a121-64c7-4514-8e16-aeba1d89ae02-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.251288 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.251269    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.251354 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.251311    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.268308 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.268280    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.268940 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.268917    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d55z8\" (UniqueName: \"kubernetes.io/projected/6785a121-64c7-4514-8e16-aeba1d89ae02-kube-api-access-d55z8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xqw44\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.338410 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.338343    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:50.651651 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:36:50.651621    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6785a121_64c7_4514_8e16_aeba1d89ae02.slice/crio-740cf49b119e248b220ec891e59a5bd990bba25d1da106891b2f711ae7feb868 WatchSource:0}: Error finding container 740cf49b119e248b220ec891e59a5bd990bba25d1da106891b2f711ae7feb868: Status 404 returned error can't find the container with id 740cf49b119e248b220ec891e59a5bd990bba25d1da106891b2f711ae7feb868
Oct 30 19:36:50.653182 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.653093    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"]
Oct 30 19:36:50.657489 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.657455    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:36:50.657603 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:50.657519    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:36:51.198417 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:51.198375    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:51.223572 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:51.223507    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44" event={"ID":"6785a121-64c7-4514-8e16-aeba1d89ae02","Type":"ContainerStarted","Data":"d50f678a81bd11d411f10c002fafedadeb8e36a2fd9bc1beeb0773051e107732"}
Oct 30 19:36:51.223793 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:51.223722    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:51.223793 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:51.223760    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44" event={"ID":"6785a121-64c7-4514-8e16-aeba1d89ae02","Type":"ContainerStarted","Data":"740cf49b119e248b220ec891e59a5bd990bba25d1da106891b2f711ae7feb868"}
Oct 30 19:36:51.224176 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:51.224146    4712 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-xqw44 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:36:51.224308 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:51.224210    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44" podUID="6785a121-64c7-4514-8e16-aeba1d89ae02" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:36:51.273169 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:51.273118    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44" podStartSLOduration=2.273103074 podStartE2EDuration="2.273103074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:51.259425484 +0000 UTC m=+964.558687615" watchObservedRunningTime="2026-10-30 19:36:51.273103074 +0000 UTC m=+964.572365204"
Oct 30 19:36:52.216205 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:52.216175    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:36:52.662921 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:52.662890    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:54.183144 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:54.183107    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:55.881048 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:55.881003    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:57.285899 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:57.285866    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:36:58.889634 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:36:58.889596    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:00.437970 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.437925    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:00.531725 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.531693    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"]
Oct 30 19:37:00.536828 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.536798    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.583706 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.583665    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"]
Oct 30 19:37:00.603969 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.603936    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-workload-certs\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.604125 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.603975    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-945bf\" (UniqueName: \"kubernetes.io/projected/7f14a509-cafc-4031-ba92-be8d6b919835-kube-api-access-945bf\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.604125 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.604013    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f14a509-cafc-4031-ba92-be8d6b919835-istio-token\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.604220 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.604159    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-istio-envoy\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.604295 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.604273    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-workload-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.604567 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.604543    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f14a509-cafc-4031-ba92-be8d6b919835-istiod-ca-cert\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.604699 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.604623    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-credential-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.604699 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.604674    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f14a509-cafc-4031-ba92-be8d6b919835-istio-podinfo\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.604839 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.604721    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-istio-data\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.706253 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.706177    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f14a509-cafc-4031-ba92-be8d6b919835-istio-podinfo\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.706253 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.706228    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-istio-data\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.706484 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.706306    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-workload-certs\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.706484 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.706324    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-945bf\" (UniqueName: \"kubernetes.io/projected/7f14a509-cafc-4031-ba92-be8d6b919835-kube-api-access-945bf\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.706484 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.706375    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f14a509-cafc-4031-ba92-be8d6b919835-istio-token\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.706484 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.706449    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-istio-envoy\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.706690 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.706541    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-workload-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.706690 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.706576    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f14a509-cafc-4031-ba92-be8d6b919835-istiod-ca-cert\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.706690 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.706622    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-credential-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.706852 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.706768    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-istio-data\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.706852 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.706814    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-workload-certs\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.707113 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.707090    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-workload-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.707375 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.707345    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-credential-socket\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.707489 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.707468    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f14a509-cafc-4031-ba92-be8d6b919835-istiod-ca-cert\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.708838 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.708807    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-istio-envoy\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.708961 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.708939    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f14a509-cafc-4031-ba92-be8d6b919835-istio-podinfo\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.728278 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.728250    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f14a509-cafc-4031-ba92-be8d6b919835-istio-token\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.728610 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.728589    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-945bf\" (UniqueName: \"kubernetes.io/projected/7f14a509-cafc-4031-ba92-be8d6b919835-kube-api-access-945bf\") pod \"e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") " pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:00.863181 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:00.863154    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:01.180972 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:37:01.180937    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f14a509_cafc_4031_ba92_be8d6b919835.slice/crio-f153c79a4f64ddc837321ea24b9ad595d5628cc1c0699d69c264b8fcb69560bd WatchSource:0}: Error finding container f153c79a4f64ddc837321ea24b9ad595d5628cc1c0699d69c264b8fcb69560bd: Status 404 returned error can't find the container with id f153c79a4f64ddc837321ea24b9ad595d5628cc1c0699d69c264b8fcb69560bd
Oct 30 19:37:01.182872 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:01.182768    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"]
Oct 30 19:37:01.183192 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:01.183156    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:37:01.183281 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:01.183219    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:37:01.183281 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:01.183247    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:37:01.317408 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:01.313712    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk" event={"ID":"7f14a509-cafc-4031-ba92-be8d6b919835","Type":"ContainerStarted","Data":"f153c79a4f64ddc837321ea24b9ad595d5628cc1c0699d69c264b8fcb69560bd"}
Oct 30 19:37:02.344392 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:02.342859    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk" event={"ID":"7f14a509-cafc-4031-ba92-be8d6b919835","Type":"ContainerStarted","Data":"ff96eb35dddd08c7c6176c419e872059818ff880b002a0e6f3b888af94187218"}
Oct 30 19:37:02.523388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:02.517811    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk" podStartSLOduration=2.517779191 podStartE2EDuration="2.517779191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:02.448522985 +0000 UTC m=+975.747785113" watchObservedRunningTime="2026-10-30 19:37:02.517779191 +0000 UTC m=+975.817041321"
Oct 30 19:37:02.684063 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:02.682839    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:02.802390 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:02.799214    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:02.866064 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:02.866030    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:37:02.866316 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:02.866292    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk" podUID="7f14a509-cafc-4031-ba92-be8d6b919835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:37:02.874520 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:02.874483    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:02.880145 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:02.880102    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:02.892391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:02.889463    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:02.950468 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:02.944439    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:03.009394 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.002067    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:03.009394 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.005437    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:03.009394 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.006449    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:03.047392 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.045448    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:03.199395 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.197451    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:03.239464 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.235903    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:03.516447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.507435    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:03.529393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.520427    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:37:03.548905 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.548862    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:37:03.549421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.549118    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:03.649516 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.649466    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:03.650964 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.650936    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:03.692529 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.692481    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:03.693299 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.693276    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:37:03.722501 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.722453    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:37:03.833444 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.800250    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:03.833444 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.830761    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:37:03.959386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.953780    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:37:03.959386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.953849    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk" podUID="7f14a509-cafc-4031-ba92-be8d6b919835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:37:03.985893 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:03.980697    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:04.040386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.039468    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:04.063496 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.042211    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:04.063496 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.042437    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:37:04.063496 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.042592    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:04.063496 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.045469    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:04.063496 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.059342    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 19:37:04.096443 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.087116    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:37:04.096443 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.087224    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:37:04.096443 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.087267    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:37:04.096443 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.087337    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:37:04.221431 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.220828    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:04.576385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.545849    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:04.576385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.546157    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:37:04.576385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.546474    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:37:04.594506 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.581882    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:37:04.594506 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.582217    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:37:04.632312 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.617668    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:37:04.649498 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.638118    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:37:04.733761 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.677086    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:37:04.814384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.702466    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:04.925450 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.737708    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:37:04.925450 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.824779    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:04.925450 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.829828    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:04.925450 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.833074    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:04.925450 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.838014    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:37:04.925450 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.890435    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:05.028429 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.943551    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:37:05.028429 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:04.995462    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:05.028429 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.022556    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:37:05.028429 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.022632    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk" podUID="7f14a509-cafc-4031-ba92-be8d6b919835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:37:05.076386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.070449    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:05.076386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.070753    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:37:05.076386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.070940    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:05.076386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.072940    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:05.098520 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.098463    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:05.281510 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.281401    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:05.635440 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.567529    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:37:05.635440 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.601626    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:05.636154 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.636109    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:05.696386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.657937    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:37:05.768389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.706889    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:05.830387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.821506    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Oct 30 19:37:05.830387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.821603    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:37:05.896484 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.888875    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:37:05.896484 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.888936    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk" podUID="7f14a509-cafc-4031-ba92-be8d6b919835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:37:05.934389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.898503    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:05.934389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.925869    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:37:05.991230 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:05.982636    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:06.008355 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.008318    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:37:06.025523 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.025477    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:37:06.027193 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.027157    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:06.058311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.027394    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:37:06.059629 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.031970    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:06.059882 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.057672    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:06.220444 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.219497    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:06.242515 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.242469    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:37:06.325853 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.257588    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:37:06.326714 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.257632    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:37:06.578081 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.577984    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:06.605447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.590353    4712 patch_prober.go:29] interesting pod/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:06.605447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.590440    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst" podUID="045af806-47d7-41b6-9cd7-ab0b9c851f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:06.606841 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.606723    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:37:06.643386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.639646    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:06.856887 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.856589    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:37:06.857651 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.857615    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:06.911179 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.911144    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:37:06.911440 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.911401    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk" podUID="7f14a509-cafc-4031-ba92-be8d6b919835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:37:06.914655 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.914620    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:06.995980 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:06.993953    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.039633 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.039591    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:37:07.233584 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.233530    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.266387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.237586    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.266387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.237658    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:37:07.302407 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.295753    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:37:07.302407 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.295838    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:37:07.333525 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.333480    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:07.334166 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.334141    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:37:07.334999 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.334974    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:37:07.371098 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.371074    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:37:07.372587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.372383    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:37:07.373647 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.373629    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:37:07.373847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.373832    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:37:07.394627 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:37:07.394603    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:37:07.499731 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.498944    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:37:07.557264 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.556894    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:37:07.560975 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.560430    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:37:07.562168 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.561208    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:37:07.562168 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.561679    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:37:07.563739 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.563723    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:37:07.565352 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.565316    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:37:07.864418 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.864325    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:37:07.864796 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:07.864411    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk" podUID="7f14a509-cafc-4031-ba92-be8d6b919835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:37:08.864553 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:08.864516    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:37:08.864966 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:08.864576    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk" podUID="7f14a509-cafc-4031-ba92-be8d6b919835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:37:09.864502 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:09.864466    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:37:09.864671 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:09.864527    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk" podUID="7f14a509-cafc-4031-ba92-be8d6b919835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:37:10.401891 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:10.401850    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:10.578030 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:10.578002    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:10.767976 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:10.767901    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:10.864279 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:10.864245    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:37:10.864509 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:10.864300    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk" podUID="7f14a509-cafc-4031-ba92-be8d6b919835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:37:10.934614 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:10.934572    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:11.085571 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:11.085479    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:11.563975 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:11.563943    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:11.787293 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:11.787255    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:11.863774 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:11.863670    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:37:11.863774 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:11.863738    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk" podUID="7f14a509-cafc-4031-ba92-be8d6b919835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:37:12.260770 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:12.260731    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:12.817571 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:12.817537    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:12.869569 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:12.869537    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:12.869968 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:12.869940    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:12.870719 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:12.870697    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:37:13.005093 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:13.005037    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:13.182568 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:13.182521    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:13.663533 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:13.663497    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:15.841687 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:15.841648    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:15.995576 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:15.995546    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:17.540851 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:17.540813    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:17.909022 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:17.908977    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:18.072643 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:18.072606    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:18.840785 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:18.840756    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:19.186618 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:19.186586    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:19.341640 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:19.341599    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:19.827732 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:19.827701    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:19.992861 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:19.992833    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:20.220631 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:20.220591    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:20.386552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:20.386506    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:21.599607 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:21.599570    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:21.890534 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:21.890502    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:22.060048 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:22.060016    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:23.185599 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:23.185537    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:24.690436 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:24.690394    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:26.184724 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:26.184665    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:27.671156 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:27.671117    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:29.059299 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:29.059267    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:30.446483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.446446    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:30.593677 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.593647    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:30.653372 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.653332    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"]
Oct 30 19:37:30.658646 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.658627    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.680516 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.680497    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"]
Oct 30 19:37:30.759051 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.758950    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:30.817904 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.817869    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.818109 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.818018    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.818109 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.818079    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.818234 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.818142    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.818234 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.818200    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.818310 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.818284    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd5jv\" (UniqueName: \"kubernetes.io/projected/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-kube-api-access-vd5jv\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.818345 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.818311    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.818345 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.818341    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.818489 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.818468    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.920332 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.920293    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.920534 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.920410    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.920534 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.920456    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.920534 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.920487    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.920534 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.920513    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.920679 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.920578    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd5jv\" (UniqueName: \"kubernetes.io/projected/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-kube-api-access-vd5jv\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.920679 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.920604    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.920679 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.920635    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.920775 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.920693    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.921449 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.921389    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-data\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.921578 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.921483    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-workload-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.921578 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.921505    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-credential-socket\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.921748 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.921709    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istiod-ca-cert\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.922146 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.922117    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-workload-certs\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.923830 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.923808    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-podinfo\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.924065 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.924042    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-envoy\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.940912 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.940878    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd5jv\" (UniqueName: \"kubernetes.io/projected/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-kube-api-access-vd5jv\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.941701 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.941675    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-token\") pod \"e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") " pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:30.973425 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:30.973397    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:31.006975 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.006911    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"]
Oct 30 19:37:31.011754 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.011705    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.036826 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.036796    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"]
Oct 30 19:37:31.140470 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.140425    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-workload-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.140655 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.140513    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-credential-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.140655 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.140643    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-workload-certs\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.140787 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.140769    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn7kl\" (UniqueName: \"kubernetes.io/projected/16005c00-9c89-4553-aa62-6d81b4243851-kube-api-access-vn7kl\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.140841 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.140826    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16005c00-9c89-4553-aa62-6d81b4243851-istiod-ca-cert\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.140906 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.140892    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16005c00-9c89-4553-aa62-6d81b4243851-istio-podinfo\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.140974 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.140960    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16005c00-9c89-4553-aa62-6d81b4243851-istio-token\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.141035 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.141020    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-istio-envoy\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.141079 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.141066    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-istio-data\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.242342 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.242310    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-workload-certs\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.242549 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.242439    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn7kl\" (UniqueName: \"kubernetes.io/projected/16005c00-9c89-4553-aa62-6d81b4243851-kube-api-access-vn7kl\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.242549 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.242491    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16005c00-9c89-4553-aa62-6d81b4243851-istiod-ca-cert\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.242674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.242555    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16005c00-9c89-4553-aa62-6d81b4243851-istio-podinfo\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.242674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.242622    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16005c00-9c89-4553-aa62-6d81b4243851-istio-token\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.242779 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.242679    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-istio-envoy\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.242779 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.242685    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-workload-certs\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.242779 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.242721    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-istio-data\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.242928 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.242798    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-workload-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.242928 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.242853    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-credential-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.243165 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.243138    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-workload-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.243251 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.243176    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-credential-socket\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.243251 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.243229    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16005c00-9c89-4553-aa62-6d81b4243851-istiod-ca-cert\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.243375 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.243327    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-istio-data\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.245085 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.245060    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-istio-envoy\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.245176 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.245147    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16005c00-9c89-4553-aa62-6d81b4243851-istio-podinfo\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.261424 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.261403    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16005c00-9c89-4553-aa62-6d81b4243851-istio-token\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.261568 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.261546    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn7kl\" (UniqueName: \"kubernetes.io/projected/16005c00-9c89-4553-aa62-6d81b4243851-kube-api-access-vn7kl\") pod \"e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") " pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.305534 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:37:31.305508    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d36e5e5_7f21_45a4_961f_61196c8a5e3e.slice/crio-fe86dedc3e50b1e9ff569b8d2739c0b9f8d5351b2f372490fdeb76116f7c72dc WatchSource:0}: Error finding container fe86dedc3e50b1e9ff569b8d2739c0b9f8d5351b2f372490fdeb76116f7c72dc: Status 404 returned error can't find the container with id fe86dedc3e50b1e9ff569b8d2739c0b9f8d5351b2f372490fdeb76116f7c72dc
Oct 30 19:37:31.306098 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.306074    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"]
Oct 30 19:37:31.308448 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.308417    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:37:31.308529 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.308497    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:37:31.308572 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.308532    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:37:31.330211 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.330187    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:31.566439 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.563787    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6" event={"ID":"3d36e5e5-7f21-45a4-961f-61196c8a5e3e","Type":"ContainerStarted","Data":"a5ef458c28ac36c782dc400e61082a796b4a76ec9b6806c8b2b952a593560559"}
Oct 30 19:37:31.566439 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.563855    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6" event={"ID":"3d36e5e5-7f21-45a4-961f-61196c8a5e3e","Type":"ContainerStarted","Data":"fe86dedc3e50b1e9ff569b8d2739c0b9f8d5351b2f372490fdeb76116f7c72dc"}
Oct 30 19:37:31.607949 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.607893    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6" podStartSLOduration=1.6078789759999998 podStartE2EDuration="1.607878976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:31.606612962 +0000 UTC m=+1004.905875091" watchObservedRunningTime="2026-10-30 19:37:31.607878976 +0000 UTC m=+1004.907141105"
Oct 30 19:37:31.661897 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:37:31.661854    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16005c00_9c89_4553_aa62_6d81b4243851.slice/crio-104c0e7e54bd2ad215b07aa69440b234a274941f18239a44bce5fe2da4655211 WatchSource:0}: Error finding container 104c0e7e54bd2ad215b07aa69440b234a274941f18239a44bce5fe2da4655211: Status 404 returned error can't find the container with id 104c0e7e54bd2ad215b07aa69440b234a274941f18239a44bce5fe2da4655211
Oct 30 19:37:31.662046 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.661989    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"]
Oct 30 19:37:31.664420 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.664386    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:37:31.664510 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.664450    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:37:31.664510 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.664483    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:37:31.808000 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.807947    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:31.949319 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.949264    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:31.974401 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:31.974375    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:32.171621 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:32.171586    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:32.324919 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:32.324830    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:32.497674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:32.497627    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:32.626446 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:32.623432    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm" event={"ID":"16005c00-9c89-4553-aa62-6d81b4243851","Type":"ContainerStarted","Data":"fe4005ee64917e85be24c507f0a180b134d3cde1e07cb1c0b4ea767d179721cf"}
Oct 30 19:37:32.626446 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:32.623495    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm" event={"ID":"16005c00-9c89-4553-aa62-6d81b4243851","Type":"ContainerStarted","Data":"104c0e7e54bd2ad215b07aa69440b234a274941f18239a44bce5fe2da4655211"}
Oct 30 19:37:32.697254 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:32.697184    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm" podStartSLOduration=2.697162906 podStartE2EDuration="2.697162906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:32.685499638 +0000 UTC m=+1005.984761769" watchObservedRunningTime="2026-10-30 19:37:32.697162906 +0000 UTC m=+1005.996425036"
Oct 30 19:37:32.976077 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:32.975994    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:37:32.976077 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:32.976057    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6" podUID="3d36e5e5-7f21-45a4-961f-61196c8a5e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:37:33.346448 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:33.342453    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:33.346448 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:33.342622    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:37:33.346448 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:33.342681    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm" podUID="16005c00-9c89-4553-aa62-6d81b4243851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 19:37:33.868526 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:33.868467    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:33.979385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:33.978780    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:37:33.979385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:33.978859    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6" podUID="3d36e5e5-7f21-45a4-961f-61196c8a5e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:37:34.028834 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:34.028798    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:34.350443 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:34.349509    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:37:34.350443 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:34.349589    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm" podUID="16005c00-9c89-4553-aa62-6d81b4243851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 19:37:34.982515 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:34.982479    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:37:34.983128 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:34.983093    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6" podUID="3d36e5e5-7f21-45a4-961f-61196c8a5e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:37:35.337651 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.337564    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:37:35.337912 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.337881    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm" podUID="16005c00-9c89-4553-aa62-6d81b4243851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 19:37:35.560850 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.560795    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:35.671137 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.671104    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"]
Oct 30 19:37:35.675512 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.675487    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:37:35.698404 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.696454    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-openshift-service-ca\") pod \"maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894\" (UID: \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:37:35.698404 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.696527    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-metrics-certs\") pod \"maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894\" (UID: \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:37:35.698404 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.696634    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-maas-api-tls\") pod \"maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894\" (UID: \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:37:35.698404 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.696667    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66d8k\" (UniqueName: \"kubernetes.io/projected/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-kube-api-access-66d8k\") pod \"maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894\" (UID: \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:37:35.768851 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.768815    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-df776e5c-serving-cert\""
Oct 30 19:37:35.781588 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.781555    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"]
Oct 30 19:37:35.782867 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.782809    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:35.798039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.798009    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-openshift-service-ca\") pod \"maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894\" (UID: \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:37:35.798170 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.798073    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-metrics-certs\") pod \"maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894\" (UID: \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:37:35.798237 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.798169    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-maas-api-tls\") pod \"maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894\" (UID: \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:37:35.798237 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.798194    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66d8k\" (UniqueName: \"kubernetes.io/projected/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-kube-api-access-66d8k\") pod \"maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894\" (UID: \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:37:35.799103 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.799082    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-openshift-service-ca\") pod \"maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894\" (UID: \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:37:35.800382 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:37:35.800352    4712 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-serving-cert: secret "maas-api-e2e-derive-df776e5c-serving-cert" not found
Oct 30 19:37:35.801021 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:37:35.800922    4712 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-maas-api-tls podName:54475dab-37b2-453c-ad3c-8c9c1cb59fdc nodeName:}" failed. No retries permitted until 2026-10-30 19:37:36.300585693 +0000 UTC m=+1009.599847807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-maas-api-tls") pod "maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894" (UID: "54475dab-37b2-453c-ad3c-8c9c1cb59fdc") : secret "maas-api-e2e-derive-df776e5c-serving-cert" not found
Oct 30 19:37:35.804132 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.804111    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-metrics-certs\") pod \"maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894\" (UID: \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:37:35.903651 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.903602    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66d8k\" (UniqueName: \"kubernetes.io/projected/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-kube-api-access-66d8k\") pod \"maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894\" (UID: \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:37:35.981203 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.980759    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:37:35.981203 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.980819    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6" podUID="3d36e5e5-7f21-45a4-961f-61196c8a5e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:37:35.986268 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:35.986227    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:36.151944 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:36.151811    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:36.331781 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:36.331692    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:37:36.332060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:36.332021    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm" podUID="16005c00-9c89-4553-aa62-6d81b4243851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 19:37:36.384338 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:37:36.384301    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:37:36.389716 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:36.389692    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-maas-api-tls\") pod \"maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894\" (UID: \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:37:36.391921 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:36.391882    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-maas-api-tls\") pod \"maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894\" (UID: \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:37:36.591167 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:36.591087    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:37:36.974515 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:36.974481    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:37:36.974661 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:36.974536    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6" podUID="3d36e5e5-7f21-45a4-961f-61196c8a5e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:37:37.091989 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:37.091945    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"]
Oct 30 19:37:37.175409 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:37.172272    4712 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:37:37.330794 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:37.330756    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:37:37.331053 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:37.331026    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm" podUID="16005c00-9c89-4553-aa62-6d81b4243851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 19:37:37.784302 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:37.784255    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894" event={"ID":"54475dab-37b2-453c-ad3c-8c9c1cb59fdc","Type":"ContainerStarted","Data":"cec3cd5b9af9393df073b1b949a47f82d803afbc0fe93f51846d02e97d1bf16d"}
Oct 30 19:37:37.784547 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:37.784329    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894" event={"ID":"54475dab-37b2-453c-ad3c-8c9c1cb59fdc","Type":"ContainerStarted","Data":"15a895ad2b4cde3dd55a6ec7374c14f5e68d90975fc7d5dae84907c0cff8a49c"}
Oct 30 19:37:37.784547 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:37.784517    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:37:37.834291 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:37.834170    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894" podStartSLOduration=2.434191625 podStartE2EDuration="2.834148786s" totalImagesPullingTime="399.957161ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:37.824803518 +0000 UTC m=+1011.124065647" watchObservedRunningTime="2026-10-30 19:37:37.834148786 +0000 UTC m=+1011.133410916"
Oct 30 19:37:37.974743 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:37.974703    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:37:37.974924 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:37.974763    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6" podUID="3d36e5e5-7f21-45a4-961f-61196c8a5e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:37:38.085464 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:38.085348    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:38.331469 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:38.331431    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 19:37:38.332124 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:38.331498    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm" podUID="16005c00-9c89-4553-aa62-6d81b4243851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 19:37:38.538617 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:38.538573    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:38.740212 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:38.740075    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:39.937385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:39.937296    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:39.983384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:39.982454    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:39.983384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:39.982520    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6" podUID="3d36e5e5-7f21-45a4-961f-61196c8a5e3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:39.998381 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:39.997179    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:40.010386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:40.001609    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:40.032386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:40.022604    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:37:40.417386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:40.414759    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:40.417386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:40.414829    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm" podUID="16005c00-9c89-4553-aa62-6d81b4243851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:40.619383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:40.618562    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:41.103436 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:41.097524    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:41.135387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:41.113967    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:41.135387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:41.115535    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": context deadline exceeded"
Oct 30 19:37:41.296390 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:41.291250    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:41.362968 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:41.362861    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:41.484384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:41.391853    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:37:41.590507 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:41.581059    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:41.659387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:41.650266    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:41.660220 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:41.660188    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:41.660469 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:41.660443    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm" podUID="16005c00-9c89-4553-aa62-6d81b4243851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:41.792084 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:41.792052    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:41.881280 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:41.881236    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:37:42.094330 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:42.094225    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:42.131649 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:42.131602    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:42.176679 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:42.176631    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:42.217472 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:42.217059    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:42.331586 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:42.324717    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:42.332383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:42.332324    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:42.350144 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:42.350051    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:37:42.965295 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:42.375920    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:42.965787 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:42.132263    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:37:42.966737 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:42.407169    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:37:42.990148 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:42.480796    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:42.997454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:42.494106    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:43.009740 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:42.657189    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:37:43.022406 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:42.657243    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:43.028802 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:42.695190    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:37:43.032496 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:42.720478    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:43.036555 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.036502    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm" podUID="16005c00-9c89-4553-aa62-6d81b4243851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:43.061439 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:42.962669    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:37:43.179484 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.179430    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:37:43.226786 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.226683    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:43.311388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.307341    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:43.317384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.314161    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:43.401616 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.401575    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:43.433583 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.433536    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:37:43.470525 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.433458    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:43.504538 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.504437    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:43.564232 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.564092    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:43.564232 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.564152    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v" podUID="7f364e92-b446-4671-8a42-573ebe98d729" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:43.619386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.608414    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:37:43.619386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.608497    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:37:43.619386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.608537    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:37:43.619386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.608574    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:37:43.619386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.608611    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:37:43.619386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.608645    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:37:43.619386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.608678    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:37:43.619386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.608713    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:37:43.619386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.608746    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:37:43.619386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.608779    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:37:43.619386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.608813    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:37:43.619386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.608845    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:37:43.711008 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.710472    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:43.929587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.927570    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:37:43.981437 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.974403    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:37:43.981437 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.974795    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:43.981437 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:43.975069    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:37:44.005213 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:44.005165    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.014640 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:44.014601    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.015145 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:44.015117    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.015777 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:44.015754    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:37:44.262478 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:44.036149    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.306796 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:44.036180    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:37:44.309913 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:44.055458    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:37:44.309913 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:44.096501    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:44.309913 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:44.309724    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm" podUID="16005c00-9c89-4553-aa62-6d81b4243851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.309913 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:44.186336    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.309913 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:44.205922    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.385446 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:44.338387    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.454434 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:44.416888    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:37:44.454434 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:44.439916    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.477488 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:44.477444    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:44.582442 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:44.564831    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.032383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.024781    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:37:45.032383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.026480    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.032383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.027166    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:37:45.032383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.027394    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:37:45.032383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.027593    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 19:37:45.091387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.088932    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.103918 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.092607    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.311454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.309839    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.311454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.311288    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.326699 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.312018    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:37:45.326699 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.312322    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:37:45.338852 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.338803    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.363779 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.363745    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:45.364043 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.364011    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm" podUID="16005c00-9c89-4553-aa62-6d81b4243851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.433382 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.417351    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.446491 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.446442    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.484388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.479757    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.755387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.730777    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:45.807388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:45.782453    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Oct 30 19:37:46.037462 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.031252    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:37:46.063388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.038765    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:46.064613 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.047065    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Oct 30 19:37:46.065234 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.047092    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:37:46.103408 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.047122    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:46.103987 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.089932    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:46.313629 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.313519    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:46.314536 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.314504    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:37:46.315277 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.315252    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:37:46.315699 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.315674    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:37:46.331467 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.331440    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:46.342472 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.342436    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:46.368635 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.368589    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:46.371583 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.371551    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc" podUID="b727df7a-56c7-46fc-974a-38aebc2593c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:46.387401 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.384490    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:46.421387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.417083    4712 patch_prober.go:29] interesting pod/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:46.421387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.417154    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm" podUID="16005c00-9c89-4553-aa62-6d81b4243851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:46.421387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.417535    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:46.468385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.455513    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:46.500431 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.497978    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:46.721883 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.721846    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:46.722184 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:46.722152    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd" podUID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:47.075241 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:47.075136    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:47.135354 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:47.134458    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:47.135354 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:47.134845    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:37:47.136246 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:47.136154    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:47.222712 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:47.222574    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:47.226690 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:47.226652    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:47.239384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:47.235841    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:47.285410 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:47.279633    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:37:47.315142 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:47.315097    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:37:47.331997 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:47.321764    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:47.387407 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:47.329471    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:47.387407 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:47.329510    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:37:47.387407 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:47.356006    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:47.418201 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:47.418156    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:47.515564 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:47.505476    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:47.515564 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:47.506229    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:48.088348 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.088301    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:48.088993 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.088962    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded"
Oct 30 19:37:48.098386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.098329    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:37:48.139495 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.137684    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:37:48.139495 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.138087    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded"
Oct 30 19:37:48.139829 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.137610    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:48.171546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.170387    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:37:48.258386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.258301    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:37:48.273391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.260301    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:48.273391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.268928    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:37:48.320378 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.311445    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:48.358426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.332948    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:48.386234 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.385428    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:48.386234 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.385713    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:37:48.386234 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.385736    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 30 19:37:48.418410 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.413912    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:37:48.421395 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.418589    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:37:48.455381 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.426100    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:37:48.455381 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.427581    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:48.455381 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.434287    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:37:48.455381 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.437557    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:37:48.461625 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.461579    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:37:48.476509 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.476476    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:48.505242 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.498181    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:37:48.513395 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.506890    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:37:48.516457 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.516431    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:48.547581 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.547563    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:37:48.547814 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.547773    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:37:48.551440 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:48.551418    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:37:49.050754 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:49.050719    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:50.698553 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:50.698517    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:53.522382 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:53.522339    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:55.237719 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:55.237687    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:56.878822 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:56.878775    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:57.222301 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:57.222214    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:57.414199 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:57.414171    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:37:58.726666 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:37:58.726632    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:00.303846 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:00.303815    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:01.750216 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:01.749109    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:03.487049 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:03.487001    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:04.749468 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:04.749437    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"]
Oct 30 19:38:04.753637 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:04.753615    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:04.771277 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:04.771251    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-46bd1516-serving-cert\""
Oct 30 19:38:04.776083 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:04.776025    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"]
Oct 30 19:38:04.849528 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:04.849498    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa3a49c-11d2-4ac2-b517-950f494bddb4-openshift-service-ca\") pod \"maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr\" (UID: \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:04.849727 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:04.849708    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fa3a49c-11d2-4ac2-b517-950f494bddb4-metrics-certs\") pod \"maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr\" (UID: \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:04.849798 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:04.849772    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fa3a49c-11d2-4ac2-b517-950f494bddb4-maas-api-tls\") pod \"maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr\" (UID: \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:04.849938 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:04.849824    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssxqz\" (UniqueName: \"kubernetes.io/projected/2fa3a49c-11d2-4ac2-b517-950f494bddb4-kube-api-access-ssxqz\") pod \"maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr\" (UID: \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:04.953528 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:04.953488    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fa3a49c-11d2-4ac2-b517-950f494bddb4-metrics-certs\") pod \"maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr\" (UID: \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:04.954038 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:04.954011    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fa3a49c-11d2-4ac2-b517-950f494bddb4-maas-api-tls\") pod \"maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr\" (UID: \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:04.954388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:04.954347    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssxqz\" (UniqueName: \"kubernetes.io/projected/2fa3a49c-11d2-4ac2-b517-950f494bddb4-kube-api-access-ssxqz\") pod \"maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr\" (UID: \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:04.954806 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:04.954772    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa3a49c-11d2-4ac2-b517-950f494bddb4-openshift-service-ca\") pod \"maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr\" (UID: \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:04.956320 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:38:04.956285    4712 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-serving-cert: secret "maas-api-e2e-ait-46bd1516-serving-cert" not found
Oct 30 19:38:04.956710 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:38:04.956599    4712 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fa3a49c-11d2-4ac2-b517-950f494bddb4-maas-api-tls podName:2fa3a49c-11d2-4ac2-b517-950f494bddb4 nodeName:}" failed. No retries permitted until 2026-10-30 19:38:05.456574389 +0000 UTC m=+1038.755836510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2fa3a49c-11d2-4ac2-b517-950f494bddb4-maas-api-tls") pod "maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr" (UID: "2fa3a49c-11d2-4ac2-b517-950f494bddb4") : secret "maas-api-e2e-ait-46bd1516-serving-cert" not found
Oct 30 19:38:04.957779 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:04.957334    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa3a49c-11d2-4ac2-b517-950f494bddb4-openshift-service-ca\") pod \"maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr\" (UID: \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:04.961136 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:04.961062    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:04.965505 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:04.965479    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fa3a49c-11d2-4ac2-b517-950f494bddb4-metrics-certs\") pod \"maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr\" (UID: \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:04.995544 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:04.995513    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssxqz\" (UniqueName: \"kubernetes.io/projected/2fa3a49c-11d2-4ac2-b517-950f494bddb4-kube-api-access-ssxqz\") pod \"maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr\" (UID: \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:05.252741 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:05.252699    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:05.462465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:05.462437    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fa3a49c-11d2-4ac2-b517-950f494bddb4-maas-api-tls\") pod \"maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr\" (UID: \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:05.464743 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:05.464717    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fa3a49c-11d2-4ac2-b517-950f494bddb4-maas-api-tls\") pod \"maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr\" (UID: \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:05.669774 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:05.669746    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:05.972102 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:38:05.972071    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa3a49c_11d2_4ac2_b517_950f494bddb4.slice/crio-37422408a0301a73f09607f53cba9f09ba11ed58667c4ae3a305725990443b07 WatchSource:0}: Error finding container 37422408a0301a73f09607f53cba9f09ba11ed58667c4ae3a305725990443b07: Status 404 returned error can't find the container with id 37422408a0301a73f09607f53cba9f09ba11ed58667c4ae3a305725990443b07
Oct 30 19:38:05.972852 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:05.972823    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"]
Oct 30 19:38:06.374739 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:38:06.374704    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:38:06.859286 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:06.859190    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:06.975139 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:06.975096    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr" event={"ID":"2fa3a49c-11d2-4ac2-b517-950f494bddb4","Type":"ContainerStarted","Data":"295975d50b68e93e21a88b176ed4b4e3eb21c3871fac55a99a325f3add6a5f58"}
Oct 30 19:38:06.975801 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:06.975776    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:06.975935 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:06.975915    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr" event={"ID":"2fa3a49c-11d2-4ac2-b517-950f494bddb4","Type":"ContainerStarted","Data":"37422408a0301a73f09607f53cba9f09ba11ed58667c4ae3a305725990443b07"}
Oct 30 19:38:07.021172 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:07.021109    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr" podStartSLOduration=2.495970479 podStartE2EDuration="3.021091582s" totalImagesPullingTime="525.121103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:07.016392909 +0000 UTC m=+1040.315655030" watchObservedRunningTime="2026-10-30 19:38:07.021091582 +0000 UTC m=+1040.320353712"
Oct 30 19:38:08.932215 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:08.932184    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:12.361841 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:12.361809    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:12.989931 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:12.989901    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:14.407100 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:14.407071    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:16.188244 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:16.188188    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:17.659994 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:17.659941    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:17.999209 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:17.999124    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:18.142810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:18.142779    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:18.764668 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:18.764632    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:18.910013 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:18.909962    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:19.978761 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:19.978727    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:21.281794 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:21.281748    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:21.427773 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:21.427734    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:21.732642 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:21.732605    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:22.546317 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:22.546281    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:23.184914 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:23.184882    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:23.972680 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:23.972632    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:24.133402 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:24.133329    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:24.472570 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:24.472533    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:24.611670 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:24.611637    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:25.160492 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:25.160462    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:26.043379 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:26.037328    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:26.301773 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:26.301685    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:27.329040 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:27.329005    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:27.923675 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:27.923641    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:28.082267 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:28.082216    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:28.251473 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:28.251395    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:28.407451 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:28.407413    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:29.245685 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:29.245656    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:29.547297 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:29.547202    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:30.393617 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:30.393585    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"]
Oct 30 19:38:30.393870 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:30.393846    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894" podUID="54475dab-37b2-453c-ad3c-8c9c1cb59fdc" containerName="maas-api" containerID="cri-o://cec3cd5b9af9393df073b1b949a47f82d803afbc0fe93f51846d02e97d1bf16d" gracePeriod=30
Oct 30 19:38:30.497561 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:30.497530    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:30.849318 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:30.849288    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:38:30.935767 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:30.935679    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-maas-api-tls\") pod \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\" (UID: \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\") "
Oct 30 19:38:30.935966 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:30.935822    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-openshift-service-ca\") pod \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\" (UID: \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\") "
Oct 30 19:38:30.935966 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:30.935877    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-kube-api-access-66d8k\" (UniqueName: \"kubernetes.io/projected/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-kube-api-access-66d8k\") pod \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\" (UID: \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\") "
Oct 30 19:38:30.935966 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:30.935923    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-metrics-certs\") pod \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\" (UID: \"54475dab-37b2-453c-ad3c-8c9c1cb59fdc\") "
Oct 30 19:38:30.936516 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:30.936346    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-openshift-service-ca" pod "54475dab-37b2-453c-ad3c-8c9c1cb59fdc" (UID: "54475dab-37b2-453c-ad3c-8c9c1cb59fdc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:30.936885 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:30.936854    4712 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-openshift-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:38:30.938789 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:30.938758    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-metrics-certs" pod "54475dab-37b2-453c-ad3c-8c9c1cb59fdc" (UID: "54475dab-37b2-453c-ad3c-8c9c1cb59fdc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:30.939087 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:30.939061    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-maas-api-tls" pod "54475dab-37b2-453c-ad3c-8c9c1cb59fdc" (UID: "54475dab-37b2-453c-ad3c-8c9c1cb59fdc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:30.939180 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:30.939145    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-kube-api-access-66d8k" pod "54475dab-37b2-453c-ad3c-8c9c1cb59fdc" (UID: "54475dab-37b2-453c-ad3c-8c9c1cb59fdc"). InnerVolumeSpecName "kube-api-access-66d8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:31.037985 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:31.037953    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-maas-api-tls\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:38:31.037985 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:31.037985    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66d8k\" (UniqueName: \"kubernetes.io/projected/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-kube-api-access-66d8k\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:38:31.038262 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:31.037999    4712 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54475dab-37b2-453c-ad3c-8c9c1cb59fdc-metrics-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:38:31.164143 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:31.164108    4712 generic.go:356] "Generic (PLEG): container finished" podID="54475dab-37b2-453c-ad3c-8c9c1cb59fdc" containerID="cec3cd5b9af9393df073b1b949a47f82d803afbc0fe93f51846d02e97d1bf16d" exitCode=0
Oct 30 19:38:31.164308 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:31.164166    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"
Oct 30 19:38:31.164308 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:31.164168    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894" event={"ID":"54475dab-37b2-453c-ad3c-8c9c1cb59fdc","Type":"ContainerDied","Data":"cec3cd5b9af9393df073b1b949a47f82d803afbc0fe93f51846d02e97d1bf16d"}
Oct 30 19:38:31.164859 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:31.164310    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894" event={"ID":"54475dab-37b2-453c-ad3c-8c9c1cb59fdc","Type":"ContainerDied","Data":"15a895ad2b4cde3dd55a6ec7374c14f5e68d90975fc7d5dae84907c0cff8a49c"}
Oct 30 19:38:31.164859 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:31.164354    4712 scope.go:122] "RemoveContainer" containerID="cec3cd5b9af9393df073b1b949a47f82d803afbc0fe93f51846d02e97d1bf16d"
Oct 30 19:38:31.176995 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:31.176973    4712 scope.go:122] "RemoveContainer" containerID="cec3cd5b9af9393df073b1b949a47f82d803afbc0fe93f51846d02e97d1bf16d"
Oct 30 19:38:31.177392 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:38:31.177310    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec3cd5b9af9393df073b1b949a47f82d803afbc0fe93f51846d02e97d1bf16d\": container with ID starting with cec3cd5b9af9393df073b1b949a47f82d803afbc0fe93f51846d02e97d1bf16d not found: ID does not exist" containerID="cec3cd5b9af9393df073b1b949a47f82d803afbc0fe93f51846d02e97d1bf16d"
Oct 30 19:38:31.177511 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:31.177350    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec3cd5b9af9393df073b1b949a47f82d803afbc0fe93f51846d02e97d1bf16d"} err="failed to get container status \"cec3cd5b9af9393df073b1b949a47f82d803afbc0fe93f51846d02e97d1bf16d\": rpc error: code = NotFound desc = could not find container \"cec3cd5b9af9393df073b1b949a47f82d803afbc0fe93f51846d02e97d1bf16d\": container with ID starting with cec3cd5b9af9393df073b1b949a47f82d803afbc0fe93f51846d02e97d1bf16d not found: ID does not exist"
Oct 30 19:38:31.218066 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:31.217984    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"]
Oct 30 19:38:31.228709 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:31.228682    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df776e5c-5b96fcb8c7-77894"]
Oct 30 19:38:31.351599 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:31.351562    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:31.852526 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:31.852493    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:32.033471 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:32.024455    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:32.194419 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:32.194385    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:32.348903 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:32.348860    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:32.853933 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:32.853897    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54475dab-37b2-453c-ad3c-8c9c1cb59fdc" path="/var/lib/kubelet/pods/54475dab-37b2-453c-ad3c-8c9c1cb59fdc/volumes"
Oct 30 19:38:33.061571 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:33.061533    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:33.221407 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:33.221348    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:33.793050 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:33.793019    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:33.953225 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:33.953195    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:34.621201 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:34.621161    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:36.061247 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:36.061208    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:36.373557 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:38:36.373524    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:38:37.627175 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:37.627136    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:39.070786 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:39.070751    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:39.315289 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:39.315225    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:39.456860 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:39.456825    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:40.496876 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:40.496834    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:42.091907 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:42.091868    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:43.249717 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:43.249685    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:43.405879 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:43.405844    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:43.630995 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:43.630961    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:43.880463 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:43.880428    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:44.891807 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:44.891777    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"]
Oct 30 19:38:44.892422 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:44.892025    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr" podUID="2fa3a49c-11d2-4ac2-b517-950f494bddb4" containerName="maas-api" containerID="cri-o://295975d50b68e93e21a88b176ed4b4e3eb21c3871fac55a99a325f3add6a5f58" gracePeriod=30
Oct 30 19:38:44.977744 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:44.977709    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:45.257737 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.257709    4712 generic.go:356] "Generic (PLEG): container finished" podID="2fa3a49c-11d2-4ac2-b517-950f494bddb4" containerID="295975d50b68e93e21a88b176ed4b4e3eb21c3871fac55a99a325f3add6a5f58" exitCode=0
Oct 30 19:38:45.257880 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.257788    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr" event={"ID":"2fa3a49c-11d2-4ac2-b517-950f494bddb4","Type":"ContainerDied","Data":"295975d50b68e93e21a88b176ed4b4e3eb21c3871fac55a99a325f3add6a5f58"}
Oct 30 19:38:45.257880 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.257840    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr" event={"ID":"2fa3a49c-11d2-4ac2-b517-950f494bddb4","Type":"ContainerDied","Data":"37422408a0301a73f09607f53cba9f09ba11ed58667c4ae3a305725990443b07"}
Oct 30 19:38:45.257880 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.257861    4712 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37422408a0301a73f09607f53cba9f09ba11ed58667c4ae3a305725990443b07"
Oct 30 19:38:45.270933 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.270902    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:45.356535 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.356498    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa3a49c-11d2-4ac2-b517-950f494bddb4-kube-api-access-ssxqz\" (UniqueName: \"kubernetes.io/projected/2fa3a49c-11d2-4ac2-b517-950f494bddb4-kube-api-access-ssxqz\") pod \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\" (UID: \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\") "
Oct 30 19:38:45.357260 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.357237    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fa3a49c-11d2-4ac2-b517-950f494bddb4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa3a49c-11d2-4ac2-b517-950f494bddb4-openshift-service-ca\") pod \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\" (UID: \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\") "
Oct 30 19:38:45.357518 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.357482    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fa3a49c-11d2-4ac2-b517-950f494bddb4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fa3a49c-11d2-4ac2-b517-950f494bddb4-maas-api-tls\") pod \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\" (UID: \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\") "
Oct 30 19:38:45.357888 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.357868    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fa3a49c-11d2-4ac2-b517-950f494bddb4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fa3a49c-11d2-4ac2-b517-950f494bddb4-metrics-certs\") pod \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\" (UID: \"2fa3a49c-11d2-4ac2-b517-950f494bddb4\") "
Oct 30 19:38:45.391651 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.387615    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa3a49c-11d2-4ac2-b517-950f494bddb4-kube-api-access-ssxqz" pod "2fa3a49c-11d2-4ac2-b517-950f494bddb4" (UID: "2fa3a49c-11d2-4ac2-b517-950f494bddb4"). InnerVolumeSpecName "kube-api-access-ssxqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:38:45.404685 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.401633    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa3a49c-11d2-4ac2-b517-950f494bddb4-openshift-service-ca" pod "2fa3a49c-11d2-4ac2-b517-950f494bddb4" (UID: "2fa3a49c-11d2-4ac2-b517-950f494bddb4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:38:45.436012 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.425341    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssxqz\" (UniqueName: \"kubernetes.io/projected/2fa3a49c-11d2-4ac2-b517-950f494bddb4-kube-api-access-ssxqz\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:38:45.436012 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.425390    4712 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa3a49c-11d2-4ac2-b517-950f494bddb4-openshift-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:38:45.447159 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.447114    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa3a49c-11d2-4ac2-b517-950f494bddb4-maas-api-tls" pod "2fa3a49c-11d2-4ac2-b517-950f494bddb4" (UID: "2fa3a49c-11d2-4ac2-b517-950f494bddb4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:45.447487 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.447446    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa3a49c-11d2-4ac2-b517-950f494bddb4-metrics-certs" pod "2fa3a49c-11d2-4ac2-b517-950f494bddb4" (UID: "2fa3a49c-11d2-4ac2-b517-950f494bddb4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:38:45.527284 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.527178    4712 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fa3a49c-11d2-4ac2-b517-950f494bddb4-metrics-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:38:45.527284 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.527209    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fa3a49c-11d2-4ac2-b517-950f494bddb4-maas-api-tls\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:38:45.701719 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.701657    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:45.895335 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:45.895289    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:46.071430 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.071393    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"]
Oct 30 19:38:46.072013 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.071994    4712 cpu_manager.go:414] "containerMap: removing container" podUID="54475dab-37b2-453c-ad3c-8c9c1cb59fdc" containerName="maas-api"
Oct 30 19:38:46.072013 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.072011    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54475dab-37b2-453c-ad3c-8c9c1cb59fdc" containerName="maas-api"
Oct 30 19:38:46.072182 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.072055    4712 cpu_manager.go:414] "containerMap: removing container" podUID="2fa3a49c-11d2-4ac2-b517-950f494bddb4" containerName="maas-api"
Oct 30 19:38:46.072182 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.072060    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa3a49c-11d2-4ac2-b517-950f494bddb4" containerName="maas-api"
Oct 30 19:38:46.072182 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.072156    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="54475dab-37b2-453c-ad3c-8c9c1cb59fdc" containerName="maas-api"
Oct 30 19:38:46.072182 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.072165    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa3a49c-11d2-4ac2-b517-950f494bddb4" containerName="maas-api"
Oct 30 19:38:46.075781 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.075764    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"
Oct 30 19:38:46.098311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.098286    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b9c43c35-serving-cert\""
Oct 30 19:38:46.108449 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.108429    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"]
Oct 30 19:38:46.136187 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.136157    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60fd7e02-44fa-4848-beec-1abb148281bf-metrics-certs\") pod \"maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr\" (UID: \"60fd7e02-44fa-4848-beec-1abb148281bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"
Oct 30 19:38:46.136323 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.136304    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60fd7e02-44fa-4848-beec-1abb148281bf-maas-api-tls\") pod \"maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr\" (UID: \"60fd7e02-44fa-4848-beec-1abb148281bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"
Oct 30 19:38:46.136461 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.136443    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60fd7e02-44fa-4848-beec-1abb148281bf-openshift-service-ca\") pod \"maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr\" (UID: \"60fd7e02-44fa-4848-beec-1abb148281bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"
Oct 30 19:38:46.136524 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.136515    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ldrv\" (UniqueName: \"kubernetes.io/projected/60fd7e02-44fa-4848-beec-1abb148281bf-kube-api-access-9ldrv\") pod \"maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr\" (UID: \"60fd7e02-44fa-4848-beec-1abb148281bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"
Oct 30 19:38:46.200061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.200003    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:46.238077 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.238042    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60fd7e02-44fa-4848-beec-1abb148281bf-openshift-service-ca\") pod \"maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr\" (UID: \"60fd7e02-44fa-4848-beec-1abb148281bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"
Oct 30 19:38:46.238204 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.238119    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ldrv\" (UniqueName: \"kubernetes.io/projected/60fd7e02-44fa-4848-beec-1abb148281bf-kube-api-access-9ldrv\") pod \"maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr\" (UID: \"60fd7e02-44fa-4848-beec-1abb148281bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"
Oct 30 19:38:46.238292 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.238275    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60fd7e02-44fa-4848-beec-1abb148281bf-metrics-certs\") pod \"maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr\" (UID: \"60fd7e02-44fa-4848-beec-1abb148281bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"
Oct 30 19:38:46.238501 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.238477    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60fd7e02-44fa-4848-beec-1abb148281bf-maas-api-tls\") pod \"maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr\" (UID: \"60fd7e02-44fa-4848-beec-1abb148281bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"
Oct 30 19:38:46.238734 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.238713    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60fd7e02-44fa-4848-beec-1abb148281bf-openshift-service-ca\") pod \"maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr\" (UID: \"60fd7e02-44fa-4848-beec-1abb148281bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"
Oct 30 19:38:46.240683 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.240656    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60fd7e02-44fa-4848-beec-1abb148281bf-metrics-certs\") pod \"maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr\" (UID: \"60fd7e02-44fa-4848-beec-1abb148281bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"
Oct 30 19:38:46.240762 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.240708    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60fd7e02-44fa-4848-beec-1abb148281bf-maas-api-tls\") pod \"maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr\" (UID: \"60fd7e02-44fa-4848-beec-1abb148281bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"
Oct 30 19:38:46.257978 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.257955    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ldrv\" (UniqueName: \"kubernetes.io/projected/60fd7e02-44fa-4848-beec-1abb148281bf-kube-api-access-9ldrv\") pod \"maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr\" (UID: \"60fd7e02-44fa-4848-beec-1abb148281bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"
Oct 30 19:38:46.263865 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.263847    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"
Oct 30 19:38:46.325380 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.322396    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"]
Oct 30 19:38:46.337153 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.337118    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-46bd1516-fd4c5bdfb-jthzr"]
Oct 30 19:38:46.343884 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.343850    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:46.390950 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.390921    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"
Oct 30 19:38:46.596118 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.596088    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:46.725908 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.725880    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"]
Oct 30 19:38:46.732539 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:38:46.732507    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60fd7e02_44fa_4848_beec_1abb148281bf.slice/crio-0e72a6e2827a6a11c4cb4d81afab7fb1777d01e49cc668e4d1c99ed6baf8e11b WatchSource:0}: Error finding container 0e72a6e2827a6a11c4cb4d81afab7fb1777d01e49cc668e4d1c99ed6baf8e11b: Status 404 returned error can't find the container with id 0e72a6e2827a6a11c4cb4d81afab7fb1777d01e49cc668e4d1c99ed6baf8e11b
Oct 30 19:38:46.853620 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:46.853587    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa3a49c-11d2-4ac2-b517-950f494bddb4" path="/var/lib/kubelet/pods/2fa3a49c-11d2-4ac2-b517-950f494bddb4/volumes"
Oct 30 19:38:47.275585 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:47.275499    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr" event={"ID":"60fd7e02-44fa-4848-beec-1abb148281bf","Type":"ContainerStarted","Data":"dafecf0d77e8e25ba9a5703274179525465579783fc7e15e356d31efcfcff404"}
Oct 30 19:38:47.276734 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:47.276698    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr" event={"ID":"60fd7e02-44fa-4848-beec-1abb148281bf","Type":"ContainerStarted","Data":"0e72a6e2827a6a11c4cb4d81afab7fb1777d01e49cc668e4d1c99ed6baf8e11b"}
Oct 30 19:38:47.276879 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:47.276754    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"
Oct 30 19:38:47.332109 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:47.332049    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr" podStartSLOduration=0.897493756 podStartE2EDuration="1.332030969s" totalImagesPullingTime="434.537213ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:47.327569212 +0000 UTC m=+1080.626831342" watchObservedRunningTime="2026-10-30 19:38:47.332030969 +0000 UTC m=+1080.631293092"
Oct 30 19:38:47.699566 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:47.699532    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:48.183270 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:48.183235    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:48.367544 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:48.367504    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:50.201690 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:50.201654    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:51.859648 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:51.859611    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:52.515031 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:52.514996    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:52.798171 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:52.798082    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:52.948207 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:52.948179    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:53.713532 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:53.713503    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:54.295798 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:54.295767    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"
Oct 30 19:38:54.532019 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:54.531984    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:54.679670 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:54.679637    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:55.579736 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:55.579696    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:57.106871 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:57.106827    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:38:58.510841 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:38:58.510803    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:00.101977 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:00.101940    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:01.916003 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:01.915963    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:03.560696 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:03.560664    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:05.492770 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:05.492733    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:06.374088 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:39:06.374051    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:39:06.808023 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:06.807939    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"]
Oct 30 19:39:06.811144 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:06.811120    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:39:06.833598 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:06.833569    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c6fd28cf-serving-cert\""
Oct 30 19:39:06.843880 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:06.843853    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"]
Oct 30 19:39:06.871301 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:06.871274    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ff505d-f27c-40d9-ac03-89e22af62055-openshift-service-ca\") pod \"maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9\" (UID: \"86ff505d-f27c-40d9-ac03-89e22af62055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:39:06.871461 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:06.871329    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wfc4\" (UniqueName: \"kubernetes.io/projected/86ff505d-f27c-40d9-ac03-89e22af62055-kube-api-access-6wfc4\") pod \"maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9\" (UID: \"86ff505d-f27c-40d9-ac03-89e22af62055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:39:06.871621 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:06.871585    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86ff505d-f27c-40d9-ac03-89e22af62055-maas-api-tls\") pod \"maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9\" (UID: \"86ff505d-f27c-40d9-ac03-89e22af62055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:39:06.871697 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:06.871679    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86ff505d-f27c-40d9-ac03-89e22af62055-metrics-certs\") pod \"maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9\" (UID: \"86ff505d-f27c-40d9-ac03-89e22af62055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:39:06.942065 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:06.942036    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:06.973319 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:06.973281    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wfc4\" (UniqueName: \"kubernetes.io/projected/86ff505d-f27c-40d9-ac03-89e22af62055-kube-api-access-6wfc4\") pod \"maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9\" (UID: \"86ff505d-f27c-40d9-ac03-89e22af62055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:39:06.973515 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:06.973462    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86ff505d-f27c-40d9-ac03-89e22af62055-maas-api-tls\") pod \"maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9\" (UID: \"86ff505d-f27c-40d9-ac03-89e22af62055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:39:06.973573 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:06.973520    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86ff505d-f27c-40d9-ac03-89e22af62055-metrics-certs\") pod \"maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9\" (UID: \"86ff505d-f27c-40d9-ac03-89e22af62055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:39:06.973755 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:06.973732    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ff505d-f27c-40d9-ac03-89e22af62055-openshift-service-ca\") pod \"maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9\" (UID: \"86ff505d-f27c-40d9-ac03-89e22af62055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:39:06.974467 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:06.974434    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ff505d-f27c-40d9-ac03-89e22af62055-openshift-service-ca\") pod \"maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9\" (UID: \"86ff505d-f27c-40d9-ac03-89e22af62055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:39:06.974577 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:39:06.974533    4712 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-serving-cert: secret "maas-api-e2e-mt-c6fd28cf-serving-cert" not found
Oct 30 19:39:06.974633 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:39:06.974617    4712 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86ff505d-f27c-40d9-ac03-89e22af62055-maas-api-tls podName:86ff505d-f27c-40d9-ac03-89e22af62055 nodeName:}" failed. No retries permitted until 2026-10-30 19:39:07.474596368 +0000 UTC m=+1100.773858481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/86ff505d-f27c-40d9-ac03-89e22af62055-maas-api-tls") pod "maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9" (UID: "86ff505d-f27c-40d9-ac03-89e22af62055") : secret "maas-api-e2e-mt-c6fd28cf-serving-cert" not found
Oct 30 19:39:06.980647 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:06.980597    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86ff505d-f27c-40d9-ac03-89e22af62055-metrics-certs\") pod \"maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9\" (UID: \"86ff505d-f27c-40d9-ac03-89e22af62055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:39:07.002543 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:07.002512    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wfc4\" (UniqueName: \"kubernetes.io/projected/86ff505d-f27c-40d9-ac03-89e22af62055-kube-api-access-6wfc4\") pod \"maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9\" (UID: \"86ff505d-f27c-40d9-ac03-89e22af62055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:39:07.111800 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:07.111712    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:07.281306 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:07.281275    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:07.481852 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:07.481816    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86ff505d-f27c-40d9-ac03-89e22af62055-maas-api-tls\") pod \"maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9\" (UID: \"86ff505d-f27c-40d9-ac03-89e22af62055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:39:07.484285 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:07.484255    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86ff505d-f27c-40d9-ac03-89e22af62055-maas-api-tls\") pod \"maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9\" (UID: \"86ff505d-f27c-40d9-ac03-89e22af62055\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:39:07.724492 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:07.724459    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:39:08.054746 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:39:08.054717    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ff505d_f27c_40d9_ac03_89e22af62055.slice/crio-6334eba62ecc1f2fbe3e3903331bd6d15b5c402246b60106b44cdbb75ce780b4 WatchSource:0}: Error finding container 6334eba62ecc1f2fbe3e3903331bd6d15b5c402246b60106b44cdbb75ce780b4: Status 404 returned error can't find the container with id 6334eba62ecc1f2fbe3e3903331bd6d15b5c402246b60106b44cdbb75ce780b4
Oct 30 19:39:08.056945 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:08.056904    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"]
Oct 30 19:39:08.448315 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:08.448279    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9" event={"ID":"86ff505d-f27c-40d9-ac03-89e22af62055","Type":"ContainerStarted","Data":"6334eba62ecc1f2fbe3e3903331bd6d15b5c402246b60106b44cdbb75ce780b4"}
Oct 30 19:39:09.100265 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:09.100231    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:09.456353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:09.456322    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9" event={"ID":"86ff505d-f27c-40d9-ac03-89e22af62055","Type":"ContainerStarted","Data":"e91e4047471ff2a2c84c68f893718b7b545608df64543ecb87846167a33d5aa0"}
Oct 30 19:39:09.456559 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:09.456437    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:39:09.499082 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:09.499034    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9" podStartSLOduration=3.164739481 podStartE2EDuration="3.499020334s" totalImagesPullingTime="334.280853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:09.496396658 +0000 UTC m=+1102.795658801" watchObservedRunningTime="2026-10-30 19:39:09.499020334 +0000 UTC m=+1102.798282464"
Oct 30 19:39:10.782917 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:10.782880    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:12.303125 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:12.302784    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:13.913046 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:13.913001    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:15.469142 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:15.469118    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:39:15.554047 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:15.554006    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:17.222656 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:17.222620    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:18.320101 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.320071    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:18.339283 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.339249    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"]
Oct 30 19:39:18.339484 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.339459    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr" podUID="60fd7e02-44fa-4848-beec-1abb148281bf" containerName="maas-api" containerID="cri-o://dafecf0d77e8e25ba9a5703274179525465579783fc7e15e356d31efcfcff404" gracePeriod=30
Oct 30 19:39:18.473194 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.473134    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:18.522106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.522070    4712 generic.go:356] "Generic (PLEG): container finished" podID="60fd7e02-44fa-4848-beec-1abb148281bf" containerID="dafecf0d77e8e25ba9a5703274179525465579783fc7e15e356d31efcfcff404" exitCode=0
Oct 30 19:39:18.522254 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.522216    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr" event={"ID":"60fd7e02-44fa-4848-beec-1abb148281bf","Type":"ContainerDied","Data":"dafecf0d77e8e25ba9a5703274179525465579783fc7e15e356d31efcfcff404"}
Oct 30 19:39:18.603951 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.603927    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"
Oct 30 19:39:18.740194 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.740164    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60fd7e02-44fa-4848-beec-1abb148281bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/60fd7e02-44fa-4848-beec-1abb148281bf-metrics-certs\") pod \"60fd7e02-44fa-4848-beec-1abb148281bf\" (UID: \"60fd7e02-44fa-4848-beec-1abb148281bf\") "
Oct 30 19:39:18.740347 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.740223    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60fd7e02-44fa-4848-beec-1abb148281bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60fd7e02-44fa-4848-beec-1abb148281bf-openshift-service-ca\") pod \"60fd7e02-44fa-4848-beec-1abb148281bf\" (UID: \"60fd7e02-44fa-4848-beec-1abb148281bf\") "
Oct 30 19:39:18.740347 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.740341    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60fd7e02-44fa-4848-beec-1abb148281bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60fd7e02-44fa-4848-beec-1abb148281bf-maas-api-tls\") pod \"60fd7e02-44fa-4848-beec-1abb148281bf\" (UID: \"60fd7e02-44fa-4848-beec-1abb148281bf\") "
Oct 30 19:39:18.740549 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.740524    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60fd7e02-44fa-4848-beec-1abb148281bf-kube-api-access-9ldrv\" (UniqueName: \"kubernetes.io/projected/60fd7e02-44fa-4848-beec-1abb148281bf-kube-api-access-9ldrv\") pod \"60fd7e02-44fa-4848-beec-1abb148281bf\" (UID: \"60fd7e02-44fa-4848-beec-1abb148281bf\") "
Oct 30 19:39:18.740632 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.740603    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60fd7e02-44fa-4848-beec-1abb148281bf-openshift-service-ca" pod "60fd7e02-44fa-4848-beec-1abb148281bf" (UID: "60fd7e02-44fa-4848-beec-1abb148281bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:18.741109 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.741087    4712 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60fd7e02-44fa-4848-beec-1abb148281bf-openshift-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:39:18.742667 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.742643    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60fd7e02-44fa-4848-beec-1abb148281bf-kube-api-access-9ldrv" pod "60fd7e02-44fa-4848-beec-1abb148281bf" (UID: "60fd7e02-44fa-4848-beec-1abb148281bf"). InnerVolumeSpecName "kube-api-access-9ldrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:18.742873 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.742857    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60fd7e02-44fa-4848-beec-1abb148281bf-metrics-certs" pod "60fd7e02-44fa-4848-beec-1abb148281bf" (UID: "60fd7e02-44fa-4848-beec-1abb148281bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:18.742936 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.742878    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60fd7e02-44fa-4848-beec-1abb148281bf-maas-api-tls" pod "60fd7e02-44fa-4848-beec-1abb148281bf" (UID: "60fd7e02-44fa-4848-beec-1abb148281bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:18.843292 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.843199    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ldrv\" (UniqueName: \"kubernetes.io/projected/60fd7e02-44fa-4848-beec-1abb148281bf-kube-api-access-9ldrv\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:39:18.843292 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.843238    4712 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60fd7e02-44fa-4848-beec-1abb148281bf-metrics-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:39:18.843292 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:18.843254    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60fd7e02-44fa-4848-beec-1abb148281bf-maas-api-tls\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:39:19.002913 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:19.002863    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:19.488652 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:19.488616    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:19.529702 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:19.529681    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"
Oct 30 19:39:19.529702 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:19.529691    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr" event={"ID":"60fd7e02-44fa-4848-beec-1abb148281bf","Type":"ContainerDied","Data":"0e72a6e2827a6a11c4cb4d81afab7fb1777d01e49cc668e4d1c99ed6baf8e11b"}
Oct 30 19:39:19.529905 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:19.529773    4712 scope.go:122] "RemoveContainer" containerID="dafecf0d77e8e25ba9a5703274179525465579783fc7e15e356d31efcfcff404"
Oct 30 19:39:19.586196 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:19.586120    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"]
Oct 30 19:39:19.598030 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:19.598001    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9c43c35-64d9754cf7-82cbr"]
Oct 30 19:39:19.665075 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:19.665036    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:20.853570 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:20.853534    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60fd7e02-44fa-4848-beec-1abb148281bf" path="/var/lib/kubelet/pods/60fd7e02-44fa-4848-beec-1abb148281bf/volumes"
Oct 30 19:39:20.941821 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:20.941790    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:21.224043 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:21.224010    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:22.032174 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:22.032145    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:22.634076 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:22.634042    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:22.794272 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:22.794220    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:22.983877 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:22.983791    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:25.257580 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:25.257543    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:27.688814 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:27.688777    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:28.159190 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:28.159148    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:28.383648 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:28.383612    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:29.321727 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:29.321618    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:29.516375 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:29.516325    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:29.839634 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:29.839602    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:30.035436 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:30.035396    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:30.828617 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:30.828582    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:30.989700 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:30.989666    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:31.765015 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:31.764985    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:32.170587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:32.170547    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:33.457075 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:33.457042    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:34.925302 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:34.925271    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:36.322143 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:36.322083    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:36.374424 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:39:36.374387    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:39:36.510044 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:36.510014    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:38.283489 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:38.283451    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:40.169953 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:40.169906    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:41.674540 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:41.674470    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:43.242473 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.242440    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"]
Oct 30 19:39:43.243070 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.243056    4712 cpu_manager.go:414] "containerMap: removing container" podUID="60fd7e02-44fa-4848-beec-1abb148281bf" containerName="maas-api"
Oct 30 19:39:43.243121 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.243071    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60fd7e02-44fa-4848-beec-1abb148281bf" containerName="maas-api"
Oct 30 19:39:43.243157 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.243152    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="60fd7e02-44fa-4848-beec-1abb148281bf" containerName="maas-api"
Oct 30 19:39:43.246906 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.246880    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"
Oct 30 19:39:43.290836 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.290802    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-3c89af-serving-cert\""
Oct 30 19:39:43.297092 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.297062    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"]
Oct 30 19:39:43.394943 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.394909    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60db5e44-82df-473c-9fc5-f7682f2c7000-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv\" (UID: \"60db5e44-82df-473c-9fc5-f7682f2c7000\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"
Oct 30 19:39:43.395160 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.395139    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kbtn\" (UniqueName: \"kubernetes.io/projected/60db5e44-82df-473c-9fc5-f7682f2c7000-kube-api-access-7kbtn\") pod \"maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv\" (UID: \"60db5e44-82df-473c-9fc5-f7682f2c7000\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"
Oct 30 19:39:43.395241 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.395193    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60db5e44-82df-473c-9fc5-f7682f2c7000-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv\" (UID: \"60db5e44-82df-473c-9fc5-f7682f2c7000\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"
Oct 30 19:39:43.395291 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.395248    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60db5e44-82df-473c-9fc5-f7682f2c7000-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv\" (UID: \"60db5e44-82df-473c-9fc5-f7682f2c7000\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"
Oct 30 19:39:43.400747 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.400721    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:43.496786 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.496693    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60db5e44-82df-473c-9fc5-f7682f2c7000-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv\" (UID: \"60db5e44-82df-473c-9fc5-f7682f2c7000\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"
Oct 30 19:39:43.496967 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.496890    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kbtn\" (UniqueName: \"kubernetes.io/projected/60db5e44-82df-473c-9fc5-f7682f2c7000-kube-api-access-7kbtn\") pod \"maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv\" (UID: \"60db5e44-82df-473c-9fc5-f7682f2c7000\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"
Oct 30 19:39:43.496967 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.496925    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60db5e44-82df-473c-9fc5-f7682f2c7000-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv\" (UID: \"60db5e44-82df-473c-9fc5-f7682f2c7000\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"
Oct 30 19:39:43.496967 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.496947    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60db5e44-82df-473c-9fc5-f7682f2c7000-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv\" (UID: \"60db5e44-82df-473c-9fc5-f7682f2c7000\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"
Oct 30 19:39:43.498475 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.498441    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60db5e44-82df-473c-9fc5-f7682f2c7000-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv\" (UID: \"60db5e44-82df-473c-9fc5-f7682f2c7000\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"
Oct 30 19:39:43.504577 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.499794    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60db5e44-82df-473c-9fc5-f7682f2c7000-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv\" (UID: \"60db5e44-82df-473c-9fc5-f7682f2c7000\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"
Oct 30 19:39:43.514840 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.514814    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60db5e44-82df-473c-9fc5-f7682f2c7000-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv\" (UID: \"60db5e44-82df-473c-9fc5-f7682f2c7000\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"
Oct 30 19:39:43.519175 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.519144    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kbtn\" (UniqueName: \"kubernetes.io/projected/60db5e44-82df-473c-9fc5-f7682f2c7000-kube-api-access-7kbtn\") pod \"maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv\" (UID: \"60db5e44-82df-473c-9fc5-f7682f2c7000\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"
Oct 30 19:39:43.562076 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.562043    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"
Oct 30 19:39:43.565099 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.565064    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:43.765737 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.765663    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:43.881058 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:39:43.881027    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60db5e44_82df_473c_9fc5_f7682f2c7000.slice/crio-bcfb523a803abf066fa95034ae76338d825f4dfd257fdac5d5ac411b8358a87e WatchSource:0}: Error finding container bcfb523a803abf066fa95034ae76338d825f4dfd257fdac5d5ac411b8358a87e: Status 404 returned error can't find the container with id bcfb523a803abf066fa95034ae76338d825f4dfd257fdac5d5ac411b8358a87e
Oct 30 19:39:43.881276 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:43.881257    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"]
Oct 30 19:39:44.725805 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:44.725758    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv" event={"ID":"60db5e44-82df-473c-9fc5-f7682f2c7000","Type":"ContainerStarted","Data":"be5e552f33a191949cd6f7405b13bb507aa4c4e490d0a894ec09d164c482f315"}
Oct 30 19:39:44.726278 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:44.725830    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv" event={"ID":"60db5e44-82df-473c-9fc5-f7682f2c7000","Type":"ContainerStarted","Data":"bcfb523a803abf066fa95034ae76338d825f4dfd257fdac5d5ac411b8358a87e"}
Oct 30 19:39:44.726278 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:44.725856    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"
Oct 30 19:39:44.768662 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:44.768614    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv" podStartSLOduration=1.378476346 podStartE2EDuration="1.768599951s" totalImagesPullingTime="390.123605ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:44.766136854 +0000 UTC m=+1138.065398994" watchObservedRunningTime="2026-10-30 19:39:44.768599951 +0000 UTC m=+1138.067862145"
Oct 30 19:39:45.401430 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:45.401394    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:47.028983 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:47.028945    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:48.531772 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:48.531727    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:50.028462 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:50.027544    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:50.737109 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:50.737084    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"
Oct 30 19:39:51.451779 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:51.451725    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:52.948415 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:52.948372    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:54.376612 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:54.376572    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:56.089870 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:56.089835    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:57.494209 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:57.494175    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:39:58.988761 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:39:58.988725    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:00.879999 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:00.879956    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:02.764554 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:02.764515    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:03.558271 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:03.558227    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:04.019757 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:04.019726    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:04.329852 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:04.329738    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:04.603686 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:04.603587    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:04.760913 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:04.760877    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:06.317848 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:06.317803    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:06.380415 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:40:06.380356    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:40:08.225023 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:08.224980    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:09.232172 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:09.232134    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:09.486263 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:09.486187    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:09.927600 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:09.927563    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:10.871319 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:10.871288    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:11.081663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:11.081627    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:11.843490 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:11.843450    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:13.488188 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:13.488155    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:15.110570 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:15.110524    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:16.247304 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:16.247271    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:16.756654 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:16.756620    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:16.961525 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:16.960488    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:17.116629 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:17.116593    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:18.668838 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:18.668810    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:20.430379 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:20.430338    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:21.397322 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:21.397285    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:21.560022 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:21.559995    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:22.121470 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:22.121435    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:23.056259 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:23.056219    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:23.221964 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:23.221924    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:24.089707 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:24.089665    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:25.753109 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:25.753077    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:25.906719 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:25.906682    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:27.383045 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:27.383007    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:28.895325 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:28.895279    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:30.407755 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:30.407720    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:31.963228 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:31.963195    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:32.230808 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:32.230724    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:32.372674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:32.372636    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:32.527341 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:32.527254    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:32.667776 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:32.667744    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:33.579524 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:33.579490    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:35.069059 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:35.069027    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:36.374392 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:40:36.374344    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:40:36.586081 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:36.586036    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:38.016776 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.016734    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"]
Oct 30 19:40:38.021174 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.021146    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:40:38.039580 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.039554    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-34bdc0-serving-cert\""
Oct 30 19:40:38.044059 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.044034    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"]
Oct 30 19:40:38.122166 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.122129    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18df8fa5-9c53-4711-8a25-2a22b76d7d51-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z\" (UID: \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:40:38.122341 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.122173    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kks72\" (UniqueName: \"kubernetes.io/projected/18df8fa5-9c53-4711-8a25-2a22b76d7d51-kube-api-access-kks72\") pod \"maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z\" (UID: \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:40:38.122341 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.122255    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18df8fa5-9c53-4711-8a25-2a22b76d7d51-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z\" (UID: \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:40:38.122572 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.122552    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18df8fa5-9c53-4711-8a25-2a22b76d7d51-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z\" (UID: \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:40:38.147747 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.147715    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:38.224420 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.224340    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18df8fa5-9c53-4711-8a25-2a22b76d7d51-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z\" (UID: \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:40:38.229248 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.229221    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kks72\" (UniqueName: \"kubernetes.io/projected/18df8fa5-9c53-4711-8a25-2a22b76d7d51-kube-api-access-kks72\") pod \"maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z\" (UID: \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:40:38.229806 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.229783    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18df8fa5-9c53-4711-8a25-2a22b76d7d51-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z\" (UID: \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:40:38.230730 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.230705    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18df8fa5-9c53-4711-8a25-2a22b76d7d51-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z\" (UID: \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:40:38.231744 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:40:38.229115    4712 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-serving-cert: secret "maas-api-e2e-shared-a-w3-34bdc0-serving-cert" not found
Oct 30 19:40:38.232083 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.231629    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18df8fa5-9c53-4711-8a25-2a22b76d7d51-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z\" (UID: \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:40:38.232254 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:40:38.232242    4712 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18df8fa5-9c53-4711-8a25-2a22b76d7d51-maas-api-tls podName:18df8fa5-9c53-4711-8a25-2a22b76d7d51 nodeName:}" failed. No retries permitted until 2026-10-30 19:40:38.732218599 +0000 UTC m=+1192.031480713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/18df8fa5-9c53-4711-8a25-2a22b76d7d51-maas-api-tls") pod "maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z" (UID: "18df8fa5-9c53-4711-8a25-2a22b76d7d51") : secret "maas-api-e2e-shared-a-w3-34bdc0-serving-cert" not found
Oct 30 19:40:38.237205 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.237165    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18df8fa5-9c53-4711-8a25-2a22b76d7d51-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z\" (UID: \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:40:38.252088 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.252060    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kks72\" (UniqueName: \"kubernetes.io/projected/18df8fa5-9c53-4711-8a25-2a22b76d7d51-kube-api-access-kks72\") pod \"maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z\" (UID: \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:40:38.370547 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.370513    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:38.532802 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.532767    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:38.715983 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.715895    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:38.741233 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.741198    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18df8fa5-9c53-4711-8a25-2a22b76d7d51-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z\" (UID: \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:40:38.743642 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.743619    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18df8fa5-9c53-4711-8a25-2a22b76d7d51-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z\" (UID: \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:40:38.953672 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:38.953644    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:40:39.262129 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:40:39.262097    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18df8fa5_9c53_4711_8a25_2a22b76d7d51.slice/crio-1fb620c8003bc4c5b52c916d0e40ccd959f3c50b8c79275d42bdbf5bcb9bdde3 WatchSource:0}: Error finding container 1fb620c8003bc4c5b52c916d0e40ccd959f3c50b8c79275d42bdbf5bcb9bdde3: Status 404 returned error can't find the container with id 1fb620c8003bc4c5b52c916d0e40ccd959f3c50b8c79275d42bdbf5bcb9bdde3
Oct 30 19:40:39.263607 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:39.263582    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"]
Oct 30 19:40:39.881057 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:39.881018    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:40.184437 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:40.184335    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z" event={"ID":"18df8fa5-9c53-4711-8a25-2a22b76d7d51","Type":"ContainerStarted","Data":"03dc28efdcd184e37a2bd35b7aa503e96975b182647ed76740d9790627ce3c41"}
Oct 30 19:40:40.184437 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:40.184422    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z" event={"ID":"18df8fa5-9c53-4711-8a25-2a22b76d7d51","Type":"ContainerStarted","Data":"1fb620c8003bc4c5b52c916d0e40ccd959f3c50b8c79275d42bdbf5bcb9bdde3"}
Oct 30 19:40:40.184639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:40.184445    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:40:40.226877 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:40.226824    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z" podStartSLOduration=2.925858926 podStartE2EDuration="3.22681173s" totalImagesPullingTime="300.952804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:40.224649614 +0000 UTC m=+1193.523911744" watchObservedRunningTime="2026-10-30 19:40:40.22681173 +0000 UTC m=+1193.526073858"
Oct 30 19:40:41.747554 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:41.747516    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:43.183580 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:43.183546    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:44.604059 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:44.603960    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:46.198127 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:46.198083    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:40:46.200814 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:46.200745    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:47.003322 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:47.003291    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:40:47.009918 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:47.009895    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:40:47.286463 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:47.286346    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:49.028102 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:49.028060    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:49.399948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:49.399903    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:49.802568 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:49.802479    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:50.920302 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:50.920265    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:51.513450 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:51.513419    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:51.988753 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:51.988715    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:52.149980 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:52.149941    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:52.720398 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:52.720349    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:54.447070 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:54.447034    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:55.983669 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:55.983632    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:56.243754 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:56.243677    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:57.589154 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:57.589080    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:57.746178 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:57.746144    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:57.777986 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:57.777949    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw"]
Oct 30 19:40:57.782585 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:57.782562    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw"
Oct 30 19:40:57.804958 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:57.804931    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw"]
Oct 30 19:40:57.811084 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:57.811056    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-67953efd\""
Oct 30 19:40:57.811202 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:57.811143    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-67953efd-dockercfg-lslns\""
Oct 30 19:40:57.859834 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:57.859774    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9wq9\" (UniqueName: \"kubernetes.io/projected/7d165954-8903-4093-ab3a-81488fe6be3e-kube-api-access-s9wq9\") pod \"payload-processing-e2e-mt-67953efd-b8797cccd-66cpw\" (UID: \"7d165954-8903-4093-ab3a-81488fe6be3e\") " pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw"
Oct 30 19:40:57.859976 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:57.859950    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d165954-8903-4093-ab3a-81488fe6be3e-plugins-config-volume\") pod \"payload-processing-e2e-mt-67953efd-b8797cccd-66cpw\" (UID: \"7d165954-8903-4093-ab3a-81488fe6be3e\") " pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw"
Oct 30 19:40:57.938419 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:57.931786    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:57.961699 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:57.961649    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9wq9\" (UniqueName: \"kubernetes.io/projected/7d165954-8903-4093-ab3a-81488fe6be3e-kube-api-access-s9wq9\") pod \"payload-processing-e2e-mt-67953efd-b8797cccd-66cpw\" (UID: \"7d165954-8903-4093-ab3a-81488fe6be3e\") " pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw"
Oct 30 19:40:57.961891 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:57.961824    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d165954-8903-4093-ab3a-81488fe6be3e-plugins-config-volume\") pod \"payload-processing-e2e-mt-67953efd-b8797cccd-66cpw\" (UID: \"7d165954-8903-4093-ab3a-81488fe6be3e\") " pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw"
Oct 30 19:40:57.962909 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:57.962879    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d165954-8903-4093-ab3a-81488fe6be3e-plugins-config-volume\") pod \"payload-processing-e2e-mt-67953efd-b8797cccd-66cpw\" (UID: \"7d165954-8903-4093-ab3a-81488fe6be3e\") " pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw"
Oct 30 19:40:57.983782 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:57.983750    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9wq9\" (UniqueName: \"kubernetes.io/projected/7d165954-8903-4093-ab3a-81488fe6be3e-kube-api-access-s9wq9\") pod \"payload-processing-e2e-mt-67953efd-b8797cccd-66cpw\" (UID: \"7d165954-8903-4093-ab3a-81488fe6be3e\") " pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw"
Oct 30 19:40:58.099072 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:58.099031    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw"
Oct 30 19:40:58.420343 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:40:58.420299    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d165954_8903_4093_ab3a_81488fe6be3e.slice/crio-26c994edc5a4967b202a648a435ef520f28a1844d29588502eb3ed6b0b885d03 WatchSource:0}: Error finding container 26c994edc5a4967b202a648a435ef520f28a1844d29588502eb3ed6b0b885d03: Status 404 returned error can't find the container with id 26c994edc5a4967b202a648a435ef520f28a1844d29588502eb3ed6b0b885d03
Oct 30 19:40:58.422053 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:58.422031    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw"]
Oct 30 19:40:58.702893 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:58.702817    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:59.336217 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.336173    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw" event={"ID":"7d165954-8903-4093-ab3a-81488fe6be3e","Type":"ContainerStarted","Data":"441137fb591c366a3f518b645716916529a8e00e214e3178cd2623290e9def95"}
Oct 30 19:40:59.336497 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.336245    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw" event={"ID":"7d165954-8903-4093-ab3a-81488fe6be3e","Type":"ContainerStarted","Data":"26c994edc5a4967b202a648a435ef520f28a1844d29588502eb3ed6b0b885d03"}
Oct 30 19:40:59.336497 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.336293    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw"
Oct 30 19:40:59.350438 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.350404    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"]
Oct 30 19:40:59.369430 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.366417    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.389383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.385941    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-workload-certs\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.389383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.386082    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-token\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.389383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.386116    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsr85\" (UniqueName: \"kubernetes.io/projected/35cbcb48-e970-48e6-a942-05cd29730d9a-kube-api-access-gsr85\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.389383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.386153    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-credential-socket\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.389383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.386178    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-envoy\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.389383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.386257    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-workload-socket\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.389383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.386286    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-data\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.389383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.386426    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cbcb48-e970-48e6-a942-05cd29730d9a-istiod-ca-cert\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.389383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.386469    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-podinfo\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.398385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.397878    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"]
Oct 30 19:40:59.455706 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.455675    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-fe4ec3-openshift-default-dockercfg-xsmld\""
Oct 30 19:40:59.461245 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.461087    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw" podStartSLOduration=2.46106879 podStartE2EDuration="2.46106879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:59.387498097 +0000 UTC m=+1212.686760227" watchObservedRunningTime="2026-10-30 19:40:59.46106879 +0000 UTC m=+1212.760330920"
Oct 30 19:40:59.466698 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.466664    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:59.495400 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.488882    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-workload-socket\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.495400 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.488926    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-data\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.495400 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.489048    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cbcb48-e970-48e6-a942-05cd29730d9a-istiod-ca-cert\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.495400 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.489084    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-podinfo\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.495400 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.489265    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-workload-certs\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.495400 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.489420    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-token\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.495400 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.489475    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsr85\" (UniqueName: \"kubernetes.io/projected/35cbcb48-e970-48e6-a942-05cd29730d9a-kube-api-access-gsr85\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.495400 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.489514    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-credential-socket\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.495400 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.489540    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-envoy\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.495400 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.492284    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-envoy\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.495400 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.492559    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-workload-socket\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.495400 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.492785    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-data\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.496172 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.496125    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cbcb48-e970-48e6-a942-05cd29730d9a-istiod-ca-cert\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.502383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.500596    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-workload-certs\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.502383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.501074    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-credential-socket\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.521382 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.513690    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-podinfo\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.527380 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.523437    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-token\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.527380 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.526460    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsr85\" (UniqueName: \"kubernetes.io/projected/35cbcb48-e970-48e6-a942-05cd29730d9a-kube-api-access-gsr85\") pod \"e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") " pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:40:59.688423 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.688391    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:40:59.693485 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:40:59.693459    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:41:00.034199 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:00.034170    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"]
Oct 30 19:41:00.034606 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:41:00.034211    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cbcb48_e970_48e6_a942_05cd29730d9a.slice/crio-7cfca795ae738866e46f1860e91e541157798955e55ca8e60da29f291424a874 WatchSource:0}: Error finding container 7cfca795ae738866e46f1860e91e541157798955e55ca8e60da29f291424a874: Status 404 returned error can't find the container with id 7cfca795ae738866e46f1860e91e541157798955e55ca8e60da29f291424a874
Oct 30 19:41:00.038126 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:00.038076    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:41:00.038252 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:00.038199    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:41:00.038324 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:00.038247    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:41:00.367597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:00.361871    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728" event={"ID":"35cbcb48-e970-48e6-a942-05cd29730d9a","Type":"ContainerStarted","Data":"43bca3a96c5a5a944f8f368ca37aa14413f576d5ae25a935c72e00d1de452ac6"}
Oct 30 19:41:00.367597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:00.361930    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728" event={"ID":"35cbcb48-e970-48e6-a942-05cd29730d9a","Type":"ContainerStarted","Data":"7cfca795ae738866e46f1860e91e541157798955e55ca8e60da29f291424a874"}
Oct 30 19:41:00.409776 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:00.409695    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728" podStartSLOduration=1.40967994 podStartE2EDuration="1.40967994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:00.406767464 +0000 UTC m=+1213.706029596" watchObservedRunningTime="2026-10-30 19:41:00.40967994 +0000 UTC m=+1213.708942068"
Oct 30 19:41:00.530698 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:00.530662    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:00.693987 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:00.693880    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:41:01.701131 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:01.701095    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:41:02.104728 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:02.104503    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:02.472386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:02.471919    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:41:02.476997 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:02.476976    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:41:02.876220 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:02.876185    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:03.391958 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:03.391903    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:03.559796 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:03.559760    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:03.729021 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:03.728948    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:03.914173 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:03.914131    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:06.005566 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:06.005530    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:06.022287 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:06.022256    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4"]
Oct 30 19:41:06.027874 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:06.027853    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4"
Oct 30 19:41:06.047909 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:06.047887    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4"]
Oct 30 19:41:06.054429 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:06.054404    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-fe4ec3\""
Oct 30 19:41:06.054631 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:06.054483    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-fe4ec3-dockercfg-ddjxk\""
Oct 30 19:41:06.082940 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:06.082912    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d985a2-697f-4a97-a50b-fdecff1d5960-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4\" (UID: \"62d985a2-697f-4a97-a50b-fdecff1d5960\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4"
Oct 30 19:41:06.083152 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:06.083119    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc4ct\" (UniqueName: \"kubernetes.io/projected/62d985a2-697f-4a97-a50b-fdecff1d5960-kube-api-access-jc4ct\") pod \"payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4\" (UID: \"62d985a2-697f-4a97-a50b-fdecff1d5960\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4"
Oct 30 19:41:06.186298 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:06.186241    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d985a2-697f-4a97-a50b-fdecff1d5960-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4\" (UID: \"62d985a2-697f-4a97-a50b-fdecff1d5960\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4"
Oct 30 19:41:06.196056 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:06.190584    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc4ct\" (UniqueName: \"kubernetes.io/projected/62d985a2-697f-4a97-a50b-fdecff1d5960-kube-api-access-jc4ct\") pod \"payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4\" (UID: \"62d985a2-697f-4a97-a50b-fdecff1d5960\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4"
Oct 30 19:41:06.196056 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:06.191886    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d985a2-697f-4a97-a50b-fdecff1d5960-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4\" (UID: \"62d985a2-697f-4a97-a50b-fdecff1d5960\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4"
Oct 30 19:41:06.203055 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:06.203026    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:06.214048 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:06.214015    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc4ct\" (UniqueName: \"kubernetes.io/projected/62d985a2-697f-4a97-a50b-fdecff1d5960-kube-api-access-jc4ct\") pod \"payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4\" (UID: \"62d985a2-697f-4a97-a50b-fdecff1d5960\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4"
Oct 30 19:41:06.345213 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:06.345120    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4"
Oct 30 19:41:06.423135 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:41:06.423101    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:41:06.637719 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:06.637686    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:06.714770 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:41:06.714745    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d985a2_697f_4a97_a50b_fdecff1d5960.slice/crio-0454df5df5f94aa2d2d8b26a75b2633304ea6c1878bb8fb58537f0ca43455cc0 WatchSource:0}: Error finding container 0454df5df5f94aa2d2d8b26a75b2633304ea6c1878bb8fb58537f0ca43455cc0: Status 404 returned error can't find the container with id 0454df5df5f94aa2d2d8b26a75b2633304ea6c1878bb8fb58537f0ca43455cc0
Oct 30 19:41:06.715254 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:06.715229    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4"]
Oct 30 19:41:07.515120 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:07.515080    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4" event={"ID":"62d985a2-697f-4a97-a50b-fdecff1d5960","Type":"ContainerStarted","Data":"e3d1ab08c62a68baefc2dbf1189768c61fcdae2dc1a18ccb79a25dd84f8a99ac"}
Oct 30 19:41:07.515603 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:07.515191    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4" event={"ID":"62d985a2-697f-4a97-a50b-fdecff1d5960","Type":"ContainerStarted","Data":"0454df5df5f94aa2d2d8b26a75b2633304ea6c1878bb8fb58537f0ca43455cc0"}
Oct 30 19:41:07.515603 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:07.515346    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4"
Oct 30 19:41:07.554810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:07.554762    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4" podStartSLOduration=2.554748921 podStartE2EDuration="2.554748921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:07.553516558 +0000 UTC m=+1220.852778686" watchObservedRunningTime="2026-10-30 19:41:07.554748921 +0000 UTC m=+1220.854011047"
Oct 30 19:41:09.343892 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:09.343859    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:10.365117 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:10.365090    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw"
Oct 30 19:41:10.985081 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:10.985028    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:11.175003 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:11.174959    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:11.344806 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:11.344710    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:11.528010 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:11.527972    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:12.710523 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:12.710493    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:12.836746 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:12.836710    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv"]
Oct 30 19:41:12.841315 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:12.841291    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv"
Oct 30 19:41:12.857847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:12.857827    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7e89da\""
Oct 30 19:41:12.863321 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:12.863298    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv"]
Oct 30 19:41:12.869989 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:12.869970    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7e89da-dockercfg-kr7sp\""
Oct 30 19:41:12.873597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:12.873578    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:12.951395 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:12.930091    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q"]
Oct 30 19:41:13.036434 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.033687    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec82e39-a787-45d5-8584-e6d0d47ae7ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv\" (UID: \"6ec82e39-a787-45d5-8584-e6d0d47ae7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv"
Oct 30 19:41:13.036434 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.033886    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkf92\" (UniqueName: \"kubernetes.io/projected/6ec82e39-a787-45d5-8584-e6d0d47ae7ff-kube-api-access-mkf92\") pod \"payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv\" (UID: \"6ec82e39-a787-45d5-8584-e6d0d47ae7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv"
Oct 30 19:41:13.116385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.105842    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q"]
Oct 30 19:41:13.116385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.105967    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q"
Oct 30 19:41:13.143409 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.139215    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec82e39-a787-45d5-8584-e6d0d47ae7ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv\" (UID: \"6ec82e39-a787-45d5-8584-e6d0d47ae7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv"
Oct 30 19:41:13.143409 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.139325    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkf92\" (UniqueName: \"kubernetes.io/projected/6ec82e39-a787-45d5-8584-e6d0d47ae7ff-kube-api-access-mkf92\") pod \"payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv\" (UID: \"6ec82e39-a787-45d5-8584-e6d0d47ae7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv"
Oct 30 19:41:13.143629 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.143530    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec82e39-a787-45d5-8584-e6d0d47ae7ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv\" (UID: \"6ec82e39-a787-45d5-8584-e6d0d47ae7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv"
Oct 30 19:41:13.170410 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.170376    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkf92\" (UniqueName: \"kubernetes.io/projected/6ec82e39-a787-45d5-8584-e6d0d47ae7ff-kube-api-access-mkf92\") pod \"payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv\" (UID: \"6ec82e39-a787-45d5-8584-e6d0d47ae7ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv"
Oct 30 19:41:13.240812 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.240771    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3dce202-6631-47f0-903d-64bb021d8011-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q\" (UID: \"c3dce202-6631-47f0-903d-64bb021d8011\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q"
Oct 30 19:41:13.241191 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.241167    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbbgj\" (UniqueName: \"kubernetes.io/projected/c3dce202-6631-47f0-903d-64bb021d8011-kube-api-access-sbbgj\") pod \"payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q\" (UID: \"c3dce202-6631-47f0-903d-64bb021d8011\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q"
Oct 30 19:41:13.342649 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.342567    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbbgj\" (UniqueName: \"kubernetes.io/projected/c3dce202-6631-47f0-903d-64bb021d8011-kube-api-access-sbbgj\") pod \"payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q\" (UID: \"c3dce202-6631-47f0-903d-64bb021d8011\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q"
Oct 30 19:41:13.342787 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.342661    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3dce202-6631-47f0-903d-64bb021d8011-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q\" (UID: \"c3dce202-6631-47f0-903d-64bb021d8011\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q"
Oct 30 19:41:13.343376 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.343343    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3dce202-6631-47f0-903d-64bb021d8011-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q\" (UID: \"c3dce202-6631-47f0-903d-64bb021d8011\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q"
Oct 30 19:41:13.361850 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.361827    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbbgj\" (UniqueName: \"kubernetes.io/projected/c3dce202-6631-47f0-903d-64bb021d8011-kube-api-access-sbbgj\") pod \"payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q\" (UID: \"c3dce202-6631-47f0-903d-64bb021d8011\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q"
Oct 30 19:41:13.444267 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.444241    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q"
Oct 30 19:41:13.456925 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.456896    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv"
Oct 30 19:41:13.776186 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:41:13.759765    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3dce202_6631_47f0_903d_64bb021d8011.slice/crio-40456fc60a959d28a49b095fb826141294e5f8c47958e2f4bc63f2ee50ad5dc3 WatchSource:0}: Error finding container 40456fc60a959d28a49b095fb826141294e5f8c47958e2f4bc63f2ee50ad5dc3: Status 404 returned error can't find the container with id 40456fc60a959d28a49b095fb826141294e5f8c47958e2f4bc63f2ee50ad5dc3
Oct 30 19:41:13.776186 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.761843    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q"]
Oct 30 19:41:13.776186 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.771237    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv"]
Oct 30 19:41:13.776186 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:41:13.773326    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec82e39_a787_45d5_8584_e6d0d47ae7ff.slice/crio-e6a06a67423206689092be2942c310cc77ac212e318b2d89d788f0c5cb208c94 WatchSource:0}: Error finding container e6a06a67423206689092be2942c310cc77ac212e318b2d89d788f0c5cb208c94: Status 404 returned error can't find the container with id e6a06a67423206689092be2942c310cc77ac212e318b2d89d788f0c5cb208c94
Oct 30 19:41:13.927558 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:13.927517    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:14.569015 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:14.568976    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv" event={"ID":"6ec82e39-a787-45d5-8584-e6d0d47ae7ff","Type":"ContainerStarted","Data":"b9fcc4a5607702b6b6e05df76d37b653a40cc850ae12c1cc61c16270316b9173"}
Oct 30 19:41:14.569208 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:14.569163    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv"
Oct 30 19:41:14.569208 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:14.569194    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv" event={"ID":"6ec82e39-a787-45d5-8584-e6d0d47ae7ff","Type":"ContainerStarted","Data":"e6a06a67423206689092be2942c310cc77ac212e318b2d89d788f0c5cb208c94"}
Oct 30 19:41:14.570856 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:14.570829    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q" event={"ID":"c3dce202-6631-47f0-903d-64bb021d8011","Type":"ContainerStarted","Data":"a0923ea20bc4d840f29dcbc0a2202611591c4ded053a4cc8dddd53eaa18eec0f"}
Oct 30 19:41:14.570982 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:14.570872    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q" event={"ID":"c3dce202-6631-47f0-903d-64bb021d8011","Type":"ContainerStarted","Data":"40456fc60a959d28a49b095fb826141294e5f8c47958e2f4bc63f2ee50ad5dc3"}
Oct 30 19:41:14.570982 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:14.570905    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q"
Oct 30 19:41:14.609910 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:14.609865    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv" podStartSLOduration=2.609853216 podStartE2EDuration="2.609853216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:14.608909158 +0000 UTC m=+1227.908171288" watchObservedRunningTime="2026-10-30 19:41:14.609853216 +0000 UTC m=+1227.909115348"
Oct 30 19:41:14.647119 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:14.647067    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q" podStartSLOduration=2.647055879 podStartE2EDuration="2.647055879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:14.646834354 +0000 UTC m=+1227.946096483" watchObservedRunningTime="2026-10-30 19:41:14.647055879 +0000 UTC m=+1227.946318007"
Oct 30 19:41:15.403297 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:15.401577    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:16.992834 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:16.992798    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"]
Oct 30 19:41:16.998031 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:16.998000    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.003557 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.003531    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:17.029941 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.029908    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-34bdc0-openshift-default-dockercfg-fh8wm\""
Oct 30 19:41:17.046920 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.046885    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"]
Oct 30 19:41:17.099507 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.099471    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-token\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.099740 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.099531    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istiod-ca-cert\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.099740 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.099683    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-envoy\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.099895 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.099786    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-workload-certs\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.099895 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.099807    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfnbc\" (UniqueName: \"kubernetes.io/projected/7bef9c13-9c33-4c00-a711-d78c2c5091cb-kube-api-access-cfnbc\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.099895 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.099870    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-podinfo\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.101072 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.099934    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-workload-socket\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.101072 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.100160    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-credential-socket\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.101072 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.100205    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-data\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.202178 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.202143    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-workload-certs\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.202178 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.202180    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfnbc\" (UniqueName: \"kubernetes.io/projected/7bef9c13-9c33-4c00-a711-d78c2c5091cb-kube-api-access-cfnbc\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.202462 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.202212    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-podinfo\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.202462 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.202280    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-workload-socket\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.202462 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.202458    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-credential-socket\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.202688 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.202501    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-data\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.202688 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.202571    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-token\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.202688 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.202626    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istiod-ca-cert\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.202688 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.202664    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-workload-certs\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.202895 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.202691    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-workload-socket\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.202895 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.202706    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-envoy\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.202895 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.202805    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-credential-socket\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.202895 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.202876    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-data\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.203301 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.203277    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istiod-ca-cert\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.204744 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.204721    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-envoy\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.204983 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.204960    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-podinfo\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.229038 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.229014    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-token\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.230429 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.230406    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfnbc\" (UniqueName: \"kubernetes.io/projected/7bef9c13-9c33-4c00-a711-d78c2c5091cb-kube-api-access-cfnbc\") pod \"e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") " pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.319036 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.318975    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:17.636029 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:41:17.635995    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bef9c13_9c33_4c00_a711_d78c2c5091cb.slice/crio-e9775410c33beb44cdc97e4ab99cef3a3b2a9749cec60dfe57fd893274a6494a WatchSource:0}: Error finding container e9775410c33beb44cdc97e4ab99cef3a3b2a9749cec60dfe57fd893274a6494a: Status 404 returned error can't find the container with id e9775410c33beb44cdc97e4ab99cef3a3b2a9749cec60dfe57fd893274a6494a
Oct 30 19:41:17.637579 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.637556    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"]
Oct 30 19:41:17.639005 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.638970    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:41:17.639088 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.639039    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:41:17.639088 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:17.639069    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:41:18.365153 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.365115    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"]
Oct 30 19:41:18.365678 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.365309    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9" podUID="86ff505d-f27c-40d9-ac03-89e22af62055" containerName="maas-api" containerID="cri-o://e91e4047471ff2a2c84c68f893718b7b545608df64543ecb87846167a33d5aa0" gracePeriod=30
Oct 30 19:41:18.447678 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.447639    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:18.536014 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.535964    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4"
Oct 30 19:41:18.658918 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.658878    4712 generic.go:356] "Generic (PLEG): container finished" podID="86ff505d-f27c-40d9-ac03-89e22af62055" containerID="e91e4047471ff2a2c84c68f893718b7b545608df64543ecb87846167a33d5aa0" exitCode=0
Oct 30 19:41:18.659124 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.659052    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9" event={"ID":"86ff505d-f27c-40d9-ac03-89e22af62055","Type":"ContainerDied","Data":"e91e4047471ff2a2c84c68f893718b7b545608df64543ecb87846167a33d5aa0"}
Oct 30 19:41:18.661848 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.661813    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh" event={"ID":"7bef9c13-9c33-4c00-a711-d78c2c5091cb","Type":"ContainerStarted","Data":"4a68538b61543ee1ba9c86c58fc034de95e53de60b123b48d32f44036946b2e5"}
Oct 30 19:41:18.661970 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.661873    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh" event={"ID":"7bef9c13-9c33-4c00-a711-d78c2c5091cb","Type":"ContainerStarted","Data":"e9775410c33beb44cdc97e4ab99cef3a3b2a9749cec60dfe57fd893274a6494a"}
Oct 30 19:41:18.765414 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.752529    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh" podStartSLOduration=2.7525069540000002 podStartE2EDuration="2.752506954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:18.727664708 +0000 UTC m=+1232.026926837" watchObservedRunningTime="2026-10-30 19:41:18.752506954 +0000 UTC m=+1232.051769084"
Oct 30 19:41:18.842196 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.842161    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:41:18.878321 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.878290    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ff505d-f27c-40d9-ac03-89e22af62055-kube-api-access-6wfc4\" (UniqueName: \"kubernetes.io/projected/86ff505d-f27c-40d9-ac03-89e22af62055-kube-api-access-6wfc4\") pod \"86ff505d-f27c-40d9-ac03-89e22af62055\" (UID: \"86ff505d-f27c-40d9-ac03-89e22af62055\") "
Oct 30 19:41:18.878516 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.878453    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86ff505d-f27c-40d9-ac03-89e22af62055-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86ff505d-f27c-40d9-ac03-89e22af62055-maas-api-tls\") pod \"86ff505d-f27c-40d9-ac03-89e22af62055\" (UID: \"86ff505d-f27c-40d9-ac03-89e22af62055\") "
Oct 30 19:41:18.878516 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.878483    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86ff505d-f27c-40d9-ac03-89e22af62055-metrics-certs\" (UniqueName: \"kubernetes.io/secret/86ff505d-f27c-40d9-ac03-89e22af62055-metrics-certs\") pod \"86ff505d-f27c-40d9-ac03-89e22af62055\" (UID: \"86ff505d-f27c-40d9-ac03-89e22af62055\") "
Oct 30 19:41:18.878604 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.878593    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86ff505d-f27c-40d9-ac03-89e22af62055-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ff505d-f27c-40d9-ac03-89e22af62055-openshift-service-ca\") pod \"86ff505d-f27c-40d9-ac03-89e22af62055\" (UID: \"86ff505d-f27c-40d9-ac03-89e22af62055\") "
Oct 30 19:41:18.879279 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.879241    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ff505d-f27c-40d9-ac03-89e22af62055-openshift-service-ca" pod "86ff505d-f27c-40d9-ac03-89e22af62055" (UID: "86ff505d-f27c-40d9-ac03-89e22af62055"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:18.879454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.879416    4712 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ff505d-f27c-40d9-ac03-89e22af62055-openshift-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:41:18.881005 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.880980    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ff505d-f27c-40d9-ac03-89e22af62055-metrics-certs" pod "86ff505d-f27c-40d9-ac03-89e22af62055" (UID: "86ff505d-f27c-40d9-ac03-89e22af62055"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:18.881121 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.881042    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ff505d-f27c-40d9-ac03-89e22af62055-kube-api-access-6wfc4" pod "86ff505d-f27c-40d9-ac03-89e22af62055" (UID: "86ff505d-f27c-40d9-ac03-89e22af62055"). InnerVolumeSpecName "kube-api-access-6wfc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:18.881236 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.881163    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ff505d-f27c-40d9-ac03-89e22af62055-maas-api-tls" pod "86ff505d-f27c-40d9-ac03-89e22af62055" (UID: "86ff505d-f27c-40d9-ac03-89e22af62055"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:41:18.980873 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.980783    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wfc4\" (UniqueName: \"kubernetes.io/projected/86ff505d-f27c-40d9-ac03-89e22af62055-kube-api-access-6wfc4\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:41:18.980873 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.980827    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86ff505d-f27c-40d9-ac03-89e22af62055-maas-api-tls\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:41:18.980873 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:18.980841    4712 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86ff505d-f27c-40d9-ac03-89e22af62055-metrics-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:41:19.245595 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:19.245499    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:19.328387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:19.327540    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:19.332384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:19.331635    4712 patch_prober.go:29] interesting pod/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:41:19.332384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:19.331690    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh" podUID="7bef9c13-9c33-4c00-a711-d78c2c5091cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:41:19.670897 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:19.670862    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"
Oct 30 19:41:19.671393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:19.670907    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9" event={"ID":"86ff505d-f27c-40d9-ac03-89e22af62055","Type":"ContainerDied","Data":"6334eba62ecc1f2fbe3e3903331bd6d15b5c402246b60106b44cdbb75ce780b4"}
Oct 30 19:41:19.671393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:19.670971    4712 scope.go:122] "RemoveContainer" containerID="e91e4047471ff2a2c84c68f893718b7b545608df64543ecb87846167a33d5aa0"
Oct 30 19:41:19.740622 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:19.740590    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"]
Oct 30 19:41:19.750238 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:19.750213    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c6fd28cf-797b5cc474-8wlb9"]
Oct 30 19:41:19.945686 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:19.944946    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:20.257901 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:20.257820    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:20.341803 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:20.341504    4712 patch_prober.go:29] interesting pod/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:41:20.341803 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:20.341564    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh" podUID="7bef9c13-9c33-4c00-a711-d78c2c5091cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:41:20.557464 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:20.557352    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:20.853852 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:20.853776    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ff505d-f27c-40d9-ac03-89e22af62055" path="/var/lib/kubelet/pods/86ff505d-f27c-40d9-ac03-89e22af62055/volumes"
Oct 30 19:41:21.323387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:21.322284    4712 patch_prober.go:29] interesting pod/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:41:21.323387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:21.322340    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh" podUID="7bef9c13-9c33-4c00-a711-d78c2c5091cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:41:22.059379 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:22.057007    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:22.319503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:22.319399    4712 patch_prober.go:29] interesting pod/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:41:22.319777 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:22.319729    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh" podUID="7bef9c13-9c33-4c00-a711-d78c2c5091cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:41:23.334522 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:23.334488    4712 patch_prober.go:29] interesting pod/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:41:23.335094 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:23.335046    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh" podUID="7bef9c13-9c33-4c00-a711-d78c2c5091cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 19:41:24.324877 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:24.324846    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:24.325211 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:24.325192    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:24.325763 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:24.325744    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:41:24.548482 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:24.548448    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:24.738477 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:24.738446    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:24.926137 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:24.926101    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:25.220770 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:25.220737    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:25.583587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:25.581350    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q"
Oct 30 19:41:25.583587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:25.581452    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv"
Oct 30 19:41:27.041992 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:27.041952    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:28.848734 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:28.848703    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:29.840222 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:29.840190    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:30.047317 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:30.047271    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:30.415631 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:30.415592    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:31.281742 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:31.281705    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:31.709429 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:31.709401    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:32.295887 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.295855    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"]
Oct 30 19:41:32.296570 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.296554    4712 cpu_manager.go:414] "containerMap: removing container" podUID="86ff505d-f27c-40d9-ac03-89e22af62055" containerName="maas-api"
Oct 30 19:41:32.296570 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.296570    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ff505d-f27c-40d9-ac03-89e22af62055" containerName="maas-api"
Oct 30 19:41:32.296709 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.296680    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ff505d-f27c-40d9-ac03-89e22af62055" containerName="maas-api"
Oct 30 19:41:32.301798 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.301763    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.333682 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.333643    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4a4e2f-openshift-default-dockercfg-9l52x\""
Oct 30 19:41:32.352792 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.352760    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"]
Oct 30 19:41:32.451290 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.451254    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istiod-ca-cert\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.451505 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.451415    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-podinfo\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.451505 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.451498    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-workload-certs\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.451649 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.451610    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-workload-socket\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.451722 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.451698    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-envoy\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.451967 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.451938    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-credential-socket\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.452068 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.451999    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg8kf\" (UniqueName: \"kubernetes.io/projected/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-kube-api-access-wg8kf\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.452068 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.452029    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-data\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.452068 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.452055    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-token\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.522058 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.522024    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:32.554109 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.554033    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istiod-ca-cert\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.554109 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.554105    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-podinfo\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.554401 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.554153    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-workload-certs\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.554401 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.554234    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-workload-socket\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.554401 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.554325    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-envoy\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.554401 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.554391    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-credential-socket\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.554625 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.554450    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg8kf\" (UniqueName: \"kubernetes.io/projected/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-kube-api-access-wg8kf\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.554625 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.554478    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-data\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.554625 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.554515    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-token\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.554770 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.554641    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-workload-certs\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.554770 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.554708    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-workload-socket\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.554876 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.554840    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istiod-ca-cert\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.554994 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.554972    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-data\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.555055 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.554989    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-credential-socket\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.556639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.556617    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-envoy\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.556957 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.556935    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-podinfo\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.576673 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.576644    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg8kf\" (UniqueName: \"kubernetes.io/projected/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-kube-api-access-wg8kf\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.577186 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.577164    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-token\") pod \"e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") " pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.625617 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.625518    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:32.703425 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.701313    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:32.955833 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:41:32.955796    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd218f368_4a8f_4b85_81ea_f5c545ebe5d5.slice/crio-692f101569fd900810bd9053fe4172ecb461c7d1061c320c25cb11245c0b5447 WatchSource:0}: Error finding container 692f101569fd900810bd9053fe4172ecb461c7d1061c320c25cb11245c0b5447: Status 404 returned error can't find the container with id 692f101569fd900810bd9053fe4172ecb461c7d1061c320c25cb11245c0b5447
Oct 30 19:41:32.956431 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.956408    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"]
Oct 30 19:41:32.958757 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.958722    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:41:32.958839 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.958789    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:41:32.958839 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:32.958821    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:41:33.167790 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:33.167760    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:33.338476 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:33.336464    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:33.651121 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:33.651092    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:33.796394 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:33.796353    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m" event={"ID":"d218f368-4a8f-4b85-81ea-f5c545ebe5d5","Type":"ContainerStarted","Data":"9dc64d824b0197845e8b72983ba7f6ee3251fff4660f1fed2486420cce8a4e93"}
Oct 30 19:41:33.796546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:33.796416    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m" event={"ID":"d218f368-4a8f-4b85-81ea-f5c545ebe5d5","Type":"ContainerStarted","Data":"692f101569fd900810bd9053fe4172ecb461c7d1061c320c25cb11245c0b5447"}
Oct 30 19:41:33.846952 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:33.846895    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m" podStartSLOduration=1.846881227 podStartE2EDuration="1.846881227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:33.843398302 +0000 UTC m=+1247.142660432" watchObservedRunningTime="2026-10-30 19:41:33.846881227 +0000 UTC m=+1247.146143357"
Oct 30 19:41:34.625942 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:34.625898    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:34.634707 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:34.633670    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:34.804832 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:34.804800    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:34.805599 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:34.805575    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:41:35.041547 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:35.041422    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:35.953407 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:35.953332    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:36.201727 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:36.201693    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:36.376339 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:41:36.376293    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:41:36.440875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:36.440847    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:36.699421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:36.699343    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:37.184713 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:37.184672    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:38.352530 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:38.352489    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:38.639207 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:38.639173    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:39.613262 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:39.613226    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:39.851584 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:39.851548    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:40.107177 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:40.107140    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:40.377746 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:40.377703    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:41.891347 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:41.891315    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:43.596496 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:43.596459    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:45.134043 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:45.133998    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:46.682935 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:46.682896    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:48.220482 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:48.220447    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:50.008607 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:50.008553    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:51.989934 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:51.989885    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:53.723222 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:53.723173    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:55.842943 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:55.842902    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:56.978740 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:56.978653    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:57.338336 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:57.338249    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:57.556452 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:57.556422    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:57.744606 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:57.744573    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:41:59.512631 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:41:59.512588    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:00.504388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:00.504346    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:01.048736 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:01.048695    4712 generic.go:356] "Generic (PLEG): container finished" podID="35cbcb48-e970-48e6-a942-05cd29730d9a" containerID="43bca3a96c5a5a944f8f368ca37aa14413f576d5ae25a935c72e00d1de452ac6" exitCode=0
Oct 30 19:42:01.049214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:01.048810    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728" event={"ID":"35cbcb48-e970-48e6-a942-05cd29730d9a","Type":"ContainerDied","Data":"43bca3a96c5a5a944f8f368ca37aa14413f576d5ae25a935c72e00d1de452ac6"}
Oct 30 19:42:01.049350 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:01.049323    4712 scope.go:122] "RemoveContainer" containerID="43bca3a96c5a5a944f8f368ca37aa14413f576d5ae25a935c72e00d1de452ac6"
Oct 30 19:42:01.051458 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:01.051417    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:42:01.052468 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:01.052434    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:42:01.052597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:01.052506    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:42:01.592519 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:01.592052    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:02.058145 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:02.058115    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728" event={"ID":"35cbcb48-e970-48e6-a942-05cd29730d9a","Type":"ContainerStarted","Data":"4c1ca5dbbd04f2b2d9073d93c768e2ad31953c04f28d53a82d26bd03cfe86c0b"}
Oct 30 19:42:02.471883 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:02.471850    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:42:02.527578 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:02.527543    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:02.693962 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:02.693930    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:42:02.695454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:02.695427    4712 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:42:02.695580 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:02.695474    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728" podUID="35cbcb48-e970-48e6-a942-05cd29730d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:42:02.725438 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:02.722823    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:03.547516 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:03.547484    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:03.694761 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:03.694729    4712 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:42:03.694994 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:03.694803    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728" podUID="35cbcb48-e970-48e6-a942-05cd29730d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:42:03.746766 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:03.746731    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:03.925266 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:03.925233    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:04.129935 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:04.129902    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:04.533234 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:04.533200    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:04.694531 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:04.694493    4712 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:42:04.695175 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:04.694557    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728" podUID="35cbcb48-e970-48e6-a942-05cd29730d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:42:05.027388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:05.023231    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:05.453387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:05.449433    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:05.694074 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:05.694040    4712 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:42:05.694419 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:05.694096    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728" podUID="35cbcb48-e970-48e6-a942-05cd29730d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:42:05.774542 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:05.772776    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:05.938970 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:05.938935    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:06.141163 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:06.141131    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:06.385900 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:42:06.385863    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:42:06.694328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:06.694286    4712 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:42:06.694549 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:06.694346    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728" podUID="35cbcb48-e970-48e6-a942-05cd29730d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:42:07.446345 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:07.446311    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:07.694686 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:07.694643    4712 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:42:07.694863 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:07.694752    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728" podUID="35cbcb48-e970-48e6-a942-05cd29730d9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:42:08.698809 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:08.698781    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:42:08.699908 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:08.699883    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:42:09.761696 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:09.761665    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:10.017346 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:10.017259    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:10.249637 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:10.249605    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:11.580027 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:11.579989    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:11.774233 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:11.774201    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:12.073750 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:12.073707    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:12.961608 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:12.961580    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:14.691975 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:14.691898    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:16.373334 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:16.373302    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:16.588894 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:16.588855    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:18.319213 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:18.319176    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:19.912163 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:19.912120    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:21.718585 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:21.718558    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:23.329091 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:23.329061    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:23.471609 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:23.471567    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:24.836563 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:24.836520    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:26.616400 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:26.616349    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:28.228923 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:28.228867    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:29.728743 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:29.728583    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:31.301298 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:31.301243    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:32.545626 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:32.545591    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:32.755150 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:32.755119    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:33.015502 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:33.015462    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:35.713419 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:35.712896    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:36.551821 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:42:36.551781    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:42:37.036480 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:37.036426    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:37.037128 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:37.037096    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:37.048380 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:37.037673    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:42:37.057381 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:37.038817    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:37.108385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:37.107938    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:37.315682 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:37.315584    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:37.316946 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:37.316911    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:37.350583 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:37.350528    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:37.358695 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:37.358647    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:37.418633 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:37.418590    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:37.447597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:37.447559    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:37.480504 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:37.480470    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:37.788388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:37.768605    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:38.063417 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:38.063316    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:38.244379 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:38.244334    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:38.431998 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:38.431964    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:38.932713 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:38.932667    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:39.182045 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:39.182011    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:40.030294 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:40.030249    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:41.611346 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:41.611269    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:43.204090 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:43.204052    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:43.399130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:43.398407    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:44.513465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:44.513428    4712 generic.go:356] "Generic (PLEG): container finished" podID="d218f368-4a8f-4b85-81ea-f5c545ebe5d5" containerID="9dc64d824b0197845e8b72983ba7f6ee3251fff4660f1fed2486420cce8a4e93" exitCode=0
Oct 30 19:42:44.513897 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:44.513497    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m" event={"ID":"d218f368-4a8f-4b85-81ea-f5c545ebe5d5","Type":"ContainerDied","Data":"9dc64d824b0197845e8b72983ba7f6ee3251fff4660f1fed2486420cce8a4e93"}
Oct 30 19:42:44.513977 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:44.513962    4712 scope.go:122] "RemoveContainer" containerID="9dc64d824b0197845e8b72983ba7f6ee3251fff4660f1fed2486420cce8a4e93"
Oct 30 19:42:44.515624 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:44.515596    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:42:44.515693 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:44.515671    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:42:44.515728 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:44.515705    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:42:44.869040 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:44.869009    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:45.059384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.045721    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:45.156890 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.156809    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j"]
Oct 30 19:42:45.173596 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.173566    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j"
Oct 30 19:42:45.181427 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.179976    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j"]
Oct 30 19:42:45.204425 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.200876    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-086f2a\""
Oct 30 19:42:45.220113 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.213401    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:45.221707 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.221680    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-086f2a-dockercfg-ql8nf\""
Oct 30 19:42:45.240141 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.240110    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn5xr\" (UniqueName: \"kubernetes.io/projected/abb8e01c-a453-450b-a7bb-260879385140-kube-api-access-hn5xr\") pod \"payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j\" (UID: \"abb8e01c-a453-450b-a7bb-260879385140\") " pod="openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j"
Oct 30 19:42:45.240576 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.240549    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb8e01c-a453-450b-a7bb-260879385140-plugins-config-volume\") pod \"payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j\" (UID: \"abb8e01c-a453-450b-a7bb-260879385140\") " pod="openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j"
Oct 30 19:42:45.347428 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.347385    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb8e01c-a453-450b-a7bb-260879385140-plugins-config-volume\") pod \"payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j\" (UID: \"abb8e01c-a453-450b-a7bb-260879385140\") " pod="openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j"
Oct 30 19:42:45.347629 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.347524    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn5xr\" (UniqueName: \"kubernetes.io/projected/abb8e01c-a453-450b-a7bb-260879385140-kube-api-access-hn5xr\") pod \"payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j\" (UID: \"abb8e01c-a453-450b-a7bb-260879385140\") " pod="openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j"
Oct 30 19:42:45.348572 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.348547    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb8e01c-a453-450b-a7bb-260879385140-plugins-config-volume\") pod \"payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j\" (UID: \"abb8e01c-a453-450b-a7bb-260879385140\") " pod="openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j"
Oct 30 19:42:45.369795 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.369758    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn5xr\" (UniqueName: \"kubernetes.io/projected/abb8e01c-a453-450b-a7bb-260879385140-kube-api-access-hn5xr\") pod \"payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j\" (UID: \"abb8e01c-a453-450b-a7bb-260879385140\") " pod="openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j"
Oct 30 19:42:45.545440 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.528089    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j"
Oct 30 19:42:45.550389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.540655    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m" event={"ID":"d218f368-4a8f-4b85-81ea-f5c545ebe5d5","Type":"ContainerStarted","Data":"de33eae9153ebe54dcba2023727c08d0bd60436aea7013a6350c8c61f900ec08"}
Oct 30 19:42:45.632384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.626695    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:42:45.641389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.636492    4712 patch_prober.go:29] interesting pod/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:42:45.641389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.636566    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m" podUID="d218f368-4a8f-4b85-81ea-f5c545ebe5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:42:45.934843 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:45.934804    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j"]
Oct 30 19:42:46.549822 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:46.549786    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j" event={"ID":"abb8e01c-a453-450b-a7bb-260879385140","Type":"ContainerStarted","Data":"3769fdea6c8d3a83ba3d1d7dcf0c7c38b00325ae84e4ad65bac9668aed5963d5"}
Oct 30 19:42:46.550218 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:46.549865    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j" event={"ID":"abb8e01c-a453-450b-a7bb-260879385140","Type":"ContainerStarted","Data":"e78cf3903411adce1ad8130746aa0cc9e0d3b867adf394689855b4221bfc1aae"}
Oct 30 19:42:46.550218 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:46.549997    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j"
Oct 30 19:42:46.592884 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:46.592836    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j" podStartSLOduration=1.5928222189999999 podStartE2EDuration="1.592822219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:46.588759376 +0000 UTC m=+1319.888021507" watchObservedRunningTime="2026-10-30 19:42:46.592822219 +0000 UTC m=+1319.892084347"
Oct 30 19:42:46.629512 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:46.629481    4712 patch_prober.go:29] interesting pod/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:42:46.629665 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:46.629534    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m" podUID="d218f368-4a8f-4b85-81ea-f5c545ebe5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:42:47.489667 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:47.489628    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:47.629805 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:47.629739    4712 patch_prober.go:29] interesting pod/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:42:47.630270 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:47.629822    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m" podUID="d218f368-4a8f-4b85-81ea-f5c545ebe5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:42:48.627385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:48.627174    4712 patch_prober.go:29] interesting pod/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:42:48.627385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:48.627233    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m" podUID="d218f368-4a8f-4b85-81ea-f5c545ebe5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:42:49.143796 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:49.143764    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:49.630771 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:49.630730    4712 patch_prober.go:29] interesting pod/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:42:49.630934 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:49.630796    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m" podUID="d218f368-4a8f-4b85-81ea-f5c545ebe5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:42:49.829388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:49.825549    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:42:50.626827 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:50.626791    4712 patch_prober.go:29] interesting pod/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:42:50.627454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:50.627424    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m" podUID="d218f368-4a8f-4b85-81ea-f5c545ebe5d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:42:50.753288 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:50.753252    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:51.631107 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:51.631076    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:42:51.632132 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:51.632109    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:42:52.750764 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:52.750727    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:54.180189 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.180159    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"]
Oct 30 19:42:54.180658 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.180389    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z" podUID="18df8fa5-9c53-4711-8a25-2a22b76d7d51" containerName="maas-api" containerID="cri-o://03dc28efdcd184e37a2bd35b7aa503e96975b182647ed76740d9790627ce3c41" gracePeriod=30
Oct 30 19:42:54.310847 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.310814    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:54.551081 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.551013    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:42:54.619195 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.618310    4712 generic.go:356] "Generic (PLEG): container finished" podID="18df8fa5-9c53-4711-8a25-2a22b76d7d51" containerID="03dc28efdcd184e37a2bd35b7aa503e96975b182647ed76740d9790627ce3c41" exitCode=0
Oct 30 19:42:54.619195 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.618394    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"
Oct 30 19:42:54.619195 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.618454    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z" event={"ID":"18df8fa5-9c53-4711-8a25-2a22b76d7d51","Type":"ContainerDied","Data":"03dc28efdcd184e37a2bd35b7aa503e96975b182647ed76740d9790627ce3c41"}
Oct 30 19:42:54.619195 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.618546    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z" event={"ID":"18df8fa5-9c53-4711-8a25-2a22b76d7d51","Type":"ContainerDied","Data":"1fb620c8003bc4c5b52c916d0e40ccd959f3c50b8c79275d42bdbf5bcb9bdde3"}
Oct 30 19:42:54.619195 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.618613    4712 scope.go:122] "RemoveContainer" containerID="03dc28efdcd184e37a2bd35b7aa503e96975b182647ed76740d9790627ce3c41"
Oct 30 19:42:54.631514 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.631494    4712 scope.go:122] "RemoveContainer" containerID="03dc28efdcd184e37a2bd35b7aa503e96975b182647ed76740d9790627ce3c41"
Oct 30 19:42:54.632062 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:42:54.632039    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03dc28efdcd184e37a2bd35b7aa503e96975b182647ed76740d9790627ce3c41\": container with ID starting with 03dc28efdcd184e37a2bd35b7aa503e96975b182647ed76740d9790627ce3c41 not found: ID does not exist" containerID="03dc28efdcd184e37a2bd35b7aa503e96975b182647ed76740d9790627ce3c41"
Oct 30 19:42:54.632160 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.632069    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03dc28efdcd184e37a2bd35b7aa503e96975b182647ed76740d9790627ce3c41"} err="failed to get container status \"03dc28efdcd184e37a2bd35b7aa503e96975b182647ed76740d9790627ce3c41\": rpc error: code = NotFound desc = could not find container \"03dc28efdcd184e37a2bd35b7aa503e96975b182647ed76740d9790627ce3c41\": container with ID starting with 03dc28efdcd184e37a2bd35b7aa503e96975b182647ed76740d9790627ce3c41 not found: ID does not exist"
Oct 30 19:42:54.653840 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.653746    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18df8fa5-9c53-4711-8a25-2a22b76d7d51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18df8fa5-9c53-4711-8a25-2a22b76d7d51-maas-api-tls\") pod \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\" (UID: \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\") "
Oct 30 19:42:54.654003 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.653924    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18df8fa5-9c53-4711-8a25-2a22b76d7d51-kube-api-access-kks72\" (UniqueName: \"kubernetes.io/projected/18df8fa5-9c53-4711-8a25-2a22b76d7d51-kube-api-access-kks72\") pod \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\" (UID: \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\") "
Oct 30 19:42:54.654225 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.654204    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18df8fa5-9c53-4711-8a25-2a22b76d7d51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18df8fa5-9c53-4711-8a25-2a22b76d7d51-metrics-certs\") pod \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\" (UID: \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\") "
Oct 30 19:42:54.654344 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.654277    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18df8fa5-9c53-4711-8a25-2a22b76d7d51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18df8fa5-9c53-4711-8a25-2a22b76d7d51-openshift-service-ca\") pod \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\" (UID: \"18df8fa5-9c53-4711-8a25-2a22b76d7d51\") "
Oct 30 19:42:54.654734 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.654705    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18df8fa5-9c53-4711-8a25-2a22b76d7d51-openshift-service-ca" pod "18df8fa5-9c53-4711-8a25-2a22b76d7d51" (UID: "18df8fa5-9c53-4711-8a25-2a22b76d7d51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:54.655151 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.655127    4712 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18df8fa5-9c53-4711-8a25-2a22b76d7d51-openshift-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.657081 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.657050    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18df8fa5-9c53-4711-8a25-2a22b76d7d51-kube-api-access-kks72" pod "18df8fa5-9c53-4711-8a25-2a22b76d7d51" (UID: "18df8fa5-9c53-4711-8a25-2a22b76d7d51"). InnerVolumeSpecName "kube-api-access-kks72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:54.657181 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.657160    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18df8fa5-9c53-4711-8a25-2a22b76d7d51-metrics-certs" pod "18df8fa5-9c53-4711-8a25-2a22b76d7d51" (UID: "18df8fa5-9c53-4711-8a25-2a22b76d7d51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:54.657254 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.657236    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18df8fa5-9c53-4711-8a25-2a22b76d7d51-maas-api-tls" pod "18df8fa5-9c53-4711-8a25-2a22b76d7d51" (UID: "18df8fa5-9c53-4711-8a25-2a22b76d7d51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:54.757381 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.757017    4712 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18df8fa5-9c53-4711-8a25-2a22b76d7d51-metrics-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.757381 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.757051    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18df8fa5-9c53-4711-8a25-2a22b76d7d51-maas-api-tls\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.757381 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.757067    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kks72\" (UniqueName: \"kubernetes.io/projected/18df8fa5-9c53-4711-8a25-2a22b76d7d51-kube-api-access-kks72\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:42:54.967216 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.967183    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"]
Oct 30 19:42:54.977816 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:54.977785    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-34bdc0-755b9bbccc-4jb5z"]
Oct 30 19:42:55.019102 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:55.019055    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:55.195097 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:55.195002    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:55.621026 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:55.620992    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:55.839661 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:55.839627    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:56.854738 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:56.854705    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18df8fa5-9c53-4711-8a25-2a22b76d7d51" path="/var/lib/kubelet/pods/18df8fa5-9c53-4711-8a25-2a22b76d7d51/volumes"
Oct 30 19:42:57.434438 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:57.434341    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:42:57.559490 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:57.559457    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j"
Oct 30 19:42:59.087859 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:42:59.087815    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:00.338097 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.338066    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:00.740210 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.740176    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:00.769152 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.769122    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"]
Oct 30 19:43:00.769893 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.769870    4712 cpu_manager.go:414] "containerMap: removing container" podUID="18df8fa5-9c53-4711-8a25-2a22b76d7d51" containerName="maas-api"
Oct 30 19:43:00.769893 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.769891    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18df8fa5-9c53-4711-8a25-2a22b76d7d51" containerName="maas-api"
Oct 30 19:43:00.770102 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.770089    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="18df8fa5-9c53-4711-8a25-2a22b76d7d51" containerName="maas-api"
Oct 30 19:43:00.773456 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.773436    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.803663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.803632    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-0c305f-openshift-default-dockercfg-kzv29\""
Oct 30 19:43:00.809819 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.809786    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"]
Oct 30 19:43:00.870834 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.870760    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-workload-certs\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.871479 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.871448    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b94r7\" (UniqueName: \"kubernetes.io/projected/b4ea703f-3d59-4204-9630-e5915516c074-kube-api-access-b94r7\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.871709 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.871689    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-workload-socket\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.871857 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.871840    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-istio-data\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.872325 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.872082    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-credential-socket\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.872524 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.872497    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-istio-envoy\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.872665 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.872575    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ea703f-3d59-4204-9630-e5915516c074-istio-podinfo\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.872735 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.872701    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ea703f-3d59-4204-9630-e5915516c074-istio-token\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.872798 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.872784    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ea703f-3d59-4204-9630-e5915516c074-istiod-ca-cert\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.943160 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.943127    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:00.979403 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.974441    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-workload-certs\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.979762 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.979715    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b94r7\" (UniqueName: \"kubernetes.io/projected/b4ea703f-3d59-4204-9630-e5915516c074-kube-api-access-b94r7\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.979961 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.979941    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-workload-socket\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.980081 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.980066    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-istio-data\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.980305 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.980290    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-credential-socket\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.980466 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.980449    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-istio-envoy\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.980633 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.980620    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ea703f-3d59-4204-9630-e5915516c074-istio-podinfo\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.980870 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.980855    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ea703f-3d59-4204-9630-e5915516c074-istio-token\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.981083 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.981068    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ea703f-3d59-4204-9630-e5915516c074-istiod-ca-cert\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.983150 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.983119    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ea703f-3d59-4204-9630-e5915516c074-istiod-ca-cert\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.985574 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.985548    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-workload-socket\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.990551 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.986866    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-credential-socket\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.992254 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.987085    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-istio-data\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.992440 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.975717    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-workload-certs\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:00.996543 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:00.996442    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-istio-envoy\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:01.007643 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:01.007619    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ea703f-3d59-4204-9630-e5915516c074-istio-token\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:01.007910 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:01.007875    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b94r7\" (UniqueName: \"kubernetes.io/projected/b4ea703f-3d59-4204-9630-e5915516c074-kube-api-access-b94r7\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:01.013920 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:01.013896    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ea703f-3d59-4204-9630-e5915516c074-istio-podinfo\") pod \"e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") " pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:01.090424 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:01.090390    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:01.206695 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:01.201910    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:01.474030 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:01.452240    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"]
Oct 30 19:43:01.474030 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:01.462211    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:43:01.474030 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:01.462433    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:43:01.474030 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:01.462610    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:43:01.679955 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:01.679918    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh" event={"ID":"b4ea703f-3d59-4204-9630-e5915516c074","Type":"ContainerStarted","Data":"7abee6196605e7ab09e2cc31da25c149f0da1578809c563848315a264d9b0ed6"}
Oct 30 19:43:01.680150 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:01.679980    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh" event={"ID":"b4ea703f-3d59-4204-9630-e5915516c074","Type":"ContainerStarted","Data":"ef55a22adfdf02dface9de9758bbff3e8fe332c66ed3b21e25fb22f1140f16ce"}
Oct 30 19:43:01.732042 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:01.731943    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh" podStartSLOduration=1.7319244390000001 podStartE2EDuration="1.731924439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:01.725827178 +0000 UTC m=+1335.025089306" watchObservedRunningTime="2026-10-30 19:43:01.731924439 +0000 UTC m=+1335.031186568"
Oct 30 19:43:02.091271 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:02.091184    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:02.097879 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:02.097784    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:02.686948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:02.686915    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:02.687968 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:02.687946    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:43:02.748632 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:02.748600    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:04.305536 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:04.305496    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:04.579209 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:04.579136    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:04.780740 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:04.780707    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:04.997543 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:04.997512    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:05.144752 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:05.144722    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"]
Oct 30 19:43:05.144954 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:05.144926    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh" podUID="7bef9c13-9c33-4c00-a711-d78c2c5091cb" containerName="istio-proxy" containerID="cri-o://4a68538b61543ee1ba9c86c58fc034de95e53de60b123b48d32f44036946b2e5" gracePeriod=30
Oct 30 19:43:05.267839 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:05.267725    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:06.168287 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:06.168234    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:06.417386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:06.399074    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:06.543283 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:43:06.543178    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:43:07.378208 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:07.378170    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:09.327507 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:09.327472    4712 patch_prober.go:29] interesting pod/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:43:09.327886 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:09.327542    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh" podUID="7bef9c13-9c33-4c00-a711-d78c2c5091cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:43:09.487679 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:09.487644    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:10.453546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.453522    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:43:10.612803 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.612769    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-token\" (UniqueName: \"kubernetes.io/projected/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-token\") pod \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") "
Oct 30 19:43:10.612960 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.612859    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-data\") pod \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") "
Oct 30 19:43:10.612960 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.612898    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-workload-socket\") pod \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") "
Oct 30 19:43:10.613081 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.612978    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istiod-ca-cert\") pod \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") "
Oct 30 19:43:10.613081 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.613062    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-workload-certs\") pod \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") "
Oct 30 19:43:10.613189 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.613136    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-credential-socket\") pod \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") "
Oct 30 19:43:10.613189 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.613161    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-data" pod "7bef9c13-9c33-4c00-a711-d78c2c5091cb" (UID: "7bef9c13-9c33-4c00-a711-d78c2c5091cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:10.613290 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.613199    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-workload-socket" pod "7bef9c13-9c33-4c00-a711-d78c2c5091cb" (UID: "7bef9c13-9c33-4c00-a711-d78c2c5091cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:10.613290 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.613202    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-podinfo\") pod \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") "
Oct 30 19:43:10.613455 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.613339    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-envoy\") pod \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") "
Oct 30 19:43:10.613455 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.613337    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-workload-certs" pod "7bef9c13-9c33-4c00-a711-d78c2c5091cb" (UID: "7bef9c13-9c33-4c00-a711-d78c2c5091cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:10.613455 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.613427    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-credential-socket" pod "7bef9c13-9c33-4c00-a711-d78c2c5091cb" (UID: "7bef9c13-9c33-4c00-a711-d78c2c5091cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:10.613639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.613449    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bef9c13-9c33-4c00-a711-d78c2c5091cb-kube-api-access-cfnbc\" (UniqueName: \"kubernetes.io/projected/7bef9c13-9c33-4c00-a711-d78c2c5091cb-kube-api-access-cfnbc\") pod \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\" (UID: \"7bef9c13-9c33-4c00-a711-d78c2c5091cb\") "
Oct 30 19:43:10.613639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.613430    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istiod-ca-cert" pod "7bef9c13-9c33-4c00-a711-d78c2c5091cb" (UID: "7bef9c13-9c33-4c00-a711-d78c2c5091cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:10.614277 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.614250    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:10.614433 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.614283    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:10.614433 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.614301    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:10.614433 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.614315    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:10.614433 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.614329    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:10.614997 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.614975    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-token" pod "7bef9c13-9c33-4c00-a711-d78c2c5091cb" (UID: "7bef9c13-9c33-4c00-a711-d78c2c5091cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:10.615301 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.615287    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-podinfo" pod "7bef9c13-9c33-4c00-a711-d78c2c5091cb" (UID: "7bef9c13-9c33-4c00-a711-d78c2c5091cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:43:10.615548 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.615526    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bef9c13-9c33-4c00-a711-d78c2c5091cb-kube-api-access-cfnbc" pod "7bef9c13-9c33-4c00-a711-d78c2c5091cb" (UID: "7bef9c13-9c33-4c00-a711-d78c2c5091cb"). InnerVolumeSpecName "kube-api-access-cfnbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:10.615624 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.615528    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-envoy" pod "7bef9c13-9c33-4c00-a711-d78c2c5091cb" (UID: "7bef9c13-9c33-4c00-a711-d78c2c5091cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:10.716268 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.716212    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:10.716268 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.716237    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:10.716268 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.716246    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfnbc\" (UniqueName: \"kubernetes.io/projected/7bef9c13-9c33-4c00-a711-d78c2c5091cb-kube-api-access-cfnbc\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:10.716268 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.716256    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bef9c13-9c33-4c00-a711-d78c2c5091cb-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:10.750864 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.750826    4712 generic.go:356] "Generic (PLEG): container finished" podID="7bef9c13-9c33-4c00-a711-d78c2c5091cb" containerID="4a68538b61543ee1ba9c86c58fc034de95e53de60b123b48d32f44036946b2e5" exitCode=0
Oct 30 19:43:10.750986 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.750874    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"
Oct 30 19:43:10.750986 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.750906    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh" event={"ID":"7bef9c13-9c33-4c00-a711-d78c2c5091cb","Type":"ContainerDied","Data":"4a68538b61543ee1ba9c86c58fc034de95e53de60b123b48d32f44036946b2e5"}
Oct 30 19:43:10.750986 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.750974    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh" event={"ID":"7bef9c13-9c33-4c00-a711-d78c2c5091cb","Type":"ContainerDied","Data":"e9775410c33beb44cdc97e4ab99cef3a3b2a9749cec60dfe57fd893274a6494a"}
Oct 30 19:43:10.751095 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.751005    4712 scope.go:122] "RemoveContainer" containerID="4a68538b61543ee1ba9c86c58fc034de95e53de60b123b48d32f44036946b2e5"
Oct 30 19:43:10.763191 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.763176    4712 scope.go:122] "RemoveContainer" containerID="4a68538b61543ee1ba9c86c58fc034de95e53de60b123b48d32f44036946b2e5"
Oct 30 19:43:10.763479 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:43:10.763452    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a68538b61543ee1ba9c86c58fc034de95e53de60b123b48d32f44036946b2e5\": container with ID starting with 4a68538b61543ee1ba9c86c58fc034de95e53de60b123b48d32f44036946b2e5 not found: ID does not exist" containerID="4a68538b61543ee1ba9c86c58fc034de95e53de60b123b48d32f44036946b2e5"
Oct 30 19:43:10.763544 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.763490    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a68538b61543ee1ba9c86c58fc034de95e53de60b123b48d32f44036946b2e5"} err="failed to get container status \"4a68538b61543ee1ba9c86c58fc034de95e53de60b123b48d32f44036946b2e5\": rpc error: code = NotFound desc = could not find container \"4a68538b61543ee1ba9c86c58fc034de95e53de60b123b48d32f44036946b2e5\": container with ID starting with 4a68538b61543ee1ba9c86c58fc034de95e53de60b123b48d32f44036946b2e5 not found: ID does not exist"
Oct 30 19:43:10.808392 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.808342    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"]
Oct 30 19:43:10.820461 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.820440    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-34bdc0-openshift-default-bb6d789bf-ltsmh"]
Oct 30 19:43:10.854879 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:10.854848    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bef9c13-9c33-4c00-a711-d78c2c5091cb" path="/var/lib/kubelet/pods/7bef9c13-9c33-4c00-a711-d78c2c5091cb/volumes"
Oct 30 19:43:11.025689 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:11.025601    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:11.180818 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:11.180774    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:12.549958 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:12.549921    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:12.746167 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:12.746129    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:13.157440 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:13.157401    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:13.542228 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:13.542146    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:13.795466 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:13.795377    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:13.990112 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:13.990073    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:14.169602 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:14.169570    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:15.276866 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:15.276833    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:16.807166 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:16.807137    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q"]
Oct 30 19:43:16.840466 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:16.840435    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv"]
Oct 30 19:43:17.005196 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:17.005161    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:17.196867 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:17.196834    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:18.688085 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:18.688050    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:20.244286 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:20.244240    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:21.777507 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:21.777441    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:21.808143 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:21.808082    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q" podUID="c3dce202-6631-47f0-903d-64bb021d8011" containerName="payload-processing" containerID="cri-o://a0923ea20bc4d840f29dcbc0a2202611591c4ded053a4cc8dddd53eaa18eec0f" gracePeriod=25
Oct 30 19:43:21.840733 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:21.840680    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv" podUID="6ec82e39-a787-45d5-8584-e6d0d47ae7ff" containerName="payload-pre-processing" containerID="cri-o://b9fcc4a5607702b6b6e05df76d37b653a40cc850ae12c1cc61c16270316b9173" gracePeriod=25
Oct 30 19:43:22.131214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.131182    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q"
Oct 30 19:43:22.162438 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.162415    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv"
Oct 30 19:43:22.247544 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.247510    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3dce202-6631-47f0-903d-64bb021d8011-kube-api-access-sbbgj\" (UniqueName: \"kubernetes.io/projected/c3dce202-6631-47f0-903d-64bb021d8011-kube-api-access-sbbgj\") pod \"c3dce202-6631-47f0-903d-64bb021d8011\" (UID: \"c3dce202-6631-47f0-903d-64bb021d8011\") "
Oct 30 19:43:22.247703 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.247626    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec82e39-a787-45d5-8584-e6d0d47ae7ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec82e39-a787-45d5-8584-e6d0d47ae7ff-plugins-config-volume\") pod \"6ec82e39-a787-45d5-8584-e6d0d47ae7ff\" (UID: \"6ec82e39-a787-45d5-8584-e6d0d47ae7ff\") "
Oct 30 19:43:22.247902 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.247878    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3dce202-6631-47f0-903d-64bb021d8011-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3dce202-6631-47f0-903d-64bb021d8011-plugins-config-volume\") pod \"c3dce202-6631-47f0-903d-64bb021d8011\" (UID: \"c3dce202-6631-47f0-903d-64bb021d8011\") "
Oct 30 19:43:22.248066 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.248025    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec82e39-a787-45d5-8584-e6d0d47ae7ff-plugins-config-volume" pod "6ec82e39-a787-45d5-8584-e6d0d47ae7ff" (UID: "6ec82e39-a787-45d5-8584-e6d0d47ae7ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:22.248157 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.248076    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec82e39-a787-45d5-8584-e6d0d47ae7ff-kube-api-access-mkf92\" (UniqueName: \"kubernetes.io/projected/6ec82e39-a787-45d5-8584-e6d0d47ae7ff-kube-api-access-mkf92\") pod \"6ec82e39-a787-45d5-8584-e6d0d47ae7ff\" (UID: \"6ec82e39-a787-45d5-8584-e6d0d47ae7ff\") "
Oct 30 19:43:22.248299 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.248264    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3dce202-6631-47f0-903d-64bb021d8011-plugins-config-volume" pod "c3dce202-6631-47f0-903d-64bb021d8011" (UID: "c3dce202-6631-47f0-903d-64bb021d8011"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:22.249145 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.249120    4712 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec82e39-a787-45d5-8584-e6d0d47ae7ff-plugins-config-volume\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:22.249243 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.249147    4712 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3dce202-6631-47f0-903d-64bb021d8011-plugins-config-volume\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:22.250275 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.250245    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec82e39-a787-45d5-8584-e6d0d47ae7ff-kube-api-access-mkf92" pod "6ec82e39-a787-45d5-8584-e6d0d47ae7ff" (UID: "6ec82e39-a787-45d5-8584-e6d0d47ae7ff"). InnerVolumeSpecName "kube-api-access-mkf92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:22.250458 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.250349    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3dce202-6631-47f0-903d-64bb021d8011-kube-api-access-sbbgj" pod "c3dce202-6631-47f0-903d-64bb021d8011" (UID: "c3dce202-6631-47f0-903d-64bb021d8011"). InnerVolumeSpecName "kube-api-access-sbbgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:22.350852 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.350785    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbbgj\" (UniqueName: \"kubernetes.io/projected/c3dce202-6631-47f0-903d-64bb021d8011-kube-api-access-sbbgj\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:22.350852 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.350812    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkf92\" (UniqueName: \"kubernetes.io/projected/6ec82e39-a787-45d5-8584-e6d0d47ae7ff-kube-api-access-mkf92\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:22.848666 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.848635    4712 generic.go:356] "Generic (PLEG): container finished" podID="6ec82e39-a787-45d5-8584-e6d0d47ae7ff" containerID="b9fcc4a5607702b6b6e05df76d37b653a40cc850ae12c1cc61c16270316b9173" exitCode=0
Oct 30 19:43:22.849125 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.848750    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv"
Oct 30 19:43:22.850395 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.850340    4712 generic.go:356] "Generic (PLEG): container finished" podID="c3dce202-6631-47f0-903d-64bb021d8011" containerID="a0923ea20bc4d840f29dcbc0a2202611591c4ded053a4cc8dddd53eaa18eec0f" exitCode=0
Oct 30 19:43:22.850501 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.850449    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q"
Oct 30 19:43:22.856949 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.856916    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv" event={"ID":"6ec82e39-a787-45d5-8584-e6d0d47ae7ff","Type":"ContainerDied","Data":"b9fcc4a5607702b6b6e05df76d37b653a40cc850ae12c1cc61c16270316b9173"}
Oct 30 19:43:22.857106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.857075    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv" event={"ID":"6ec82e39-a787-45d5-8584-e6d0d47ae7ff","Type":"ContainerDied","Data":"e6a06a67423206689092be2942c310cc77ac212e318b2d89d788f0c5cb208c94"}
Oct 30 19:43:22.857185 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.857126    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q" event={"ID":"c3dce202-6631-47f0-903d-64bb021d8011","Type":"ContainerDied","Data":"a0923ea20bc4d840f29dcbc0a2202611591c4ded053a4cc8dddd53eaa18eec0f"}
Oct 30 19:43:22.857185 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.857143    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q" event={"ID":"c3dce202-6631-47f0-903d-64bb021d8011","Type":"ContainerDied","Data":"40456fc60a959d28a49b095fb826141294e5f8c47958e2f4bc63f2ee50ad5dc3"}
Oct 30 19:43:22.857185 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.857162    4712 scope.go:122] "RemoveContainer" containerID="b9fcc4a5607702b6b6e05df76d37b653a40cc850ae12c1cc61c16270316b9173"
Oct 30 19:43:22.870426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.870409    4712 scope.go:122] "RemoveContainer" containerID="b9fcc4a5607702b6b6e05df76d37b653a40cc850ae12c1cc61c16270316b9173"
Oct 30 19:43:22.870707 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:43:22.870686    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9fcc4a5607702b6b6e05df76d37b653a40cc850ae12c1cc61c16270316b9173\": container with ID starting with b9fcc4a5607702b6b6e05df76d37b653a40cc850ae12c1cc61c16270316b9173 not found: ID does not exist" containerID="b9fcc4a5607702b6b6e05df76d37b653a40cc850ae12c1cc61c16270316b9173"
Oct 30 19:43:22.870779 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.870717    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9fcc4a5607702b6b6e05df76d37b653a40cc850ae12c1cc61c16270316b9173"} err="failed to get container status \"b9fcc4a5607702b6b6e05df76d37b653a40cc850ae12c1cc61c16270316b9173\": rpc error: code = NotFound desc = could not find container \"b9fcc4a5607702b6b6e05df76d37b653a40cc850ae12c1cc61c16270316b9173\": container with ID starting with b9fcc4a5607702b6b6e05df76d37b653a40cc850ae12c1cc61c16270316b9173 not found: ID does not exist"
Oct 30 19:43:22.870779 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.870740    4712 scope.go:122] "RemoveContainer" containerID="a0923ea20bc4d840f29dcbc0a2202611591c4ded053a4cc8dddd53eaa18eec0f"
Oct 30 19:43:22.879685 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.879666    4712 scope.go:122] "RemoveContainer" containerID="a0923ea20bc4d840f29dcbc0a2202611591c4ded053a4cc8dddd53eaa18eec0f"
Oct 30 19:43:22.879962 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:43:22.879939    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0923ea20bc4d840f29dcbc0a2202611591c4ded053a4cc8dddd53eaa18eec0f\": container with ID starting with a0923ea20bc4d840f29dcbc0a2202611591c4ded053a4cc8dddd53eaa18eec0f not found: ID does not exist" containerID="a0923ea20bc4d840f29dcbc0a2202611591c4ded053a4cc8dddd53eaa18eec0f"
Oct 30 19:43:22.880007 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.879970    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0923ea20bc4d840f29dcbc0a2202611591c4ded053a4cc8dddd53eaa18eec0f"} err="failed to get container status \"a0923ea20bc4d840f29dcbc0a2202611591c4ded053a4cc8dddd53eaa18eec0f\": rpc error: code = NotFound desc = could not find container \"a0923ea20bc4d840f29dcbc0a2202611591c4ded053a4cc8dddd53eaa18eec0f\": container with ID starting with a0923ea20bc4d840f29dcbc0a2202611591c4ded053a4cc8dddd53eaa18eec0f not found: ID does not exist"
Oct 30 19:43:22.905052 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.905020    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv"]
Oct 30 19:43:22.913000 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.912973    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7e89da-b7fbfc95-cm9mv"]
Oct 30 19:43:22.955639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.955606    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q"]
Oct 30 19:43:22.964548 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:22.964518    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7e89da-8669c7db68-6265q"]
Oct 30 19:43:23.388876 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:23.388835    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:24.854637 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:24.854607    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec82e39-a787-45d5-8584-e6d0d47ae7ff" path="/var/lib/kubelet/pods/6ec82e39-a787-45d5-8584-e6d0d47ae7ff/volumes"
Oct 30 19:43:24.855049 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:24.855034    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3dce202-6631-47f0-903d-64bb021d8011" path="/var/lib/kubelet/pods/c3dce202-6631-47f0-903d-64bb021d8011/volumes"
Oct 30 19:43:24.906748 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:24.906715    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:25.958897 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:25.958866    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:26.594782 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:26.594741    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:27.866747 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:27.866717    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:28.182492 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:28.182416    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:28.408145 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:28.408098    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:28.767436 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:28.767407    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:30.090675 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:30.090639    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:31.787146 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:31.787083    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:32.526768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:32.526731    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:32.712995 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:32.712941    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:32.892956 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:32.892923    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:33.840746 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:33.840707    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:34.026630 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:34.026599    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:34.751810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:34.751779    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:34.949972 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:34.949938    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:35.436639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:35.436610    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:35.945772 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:35.945740    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:36.132103 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:36.132051    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:36.314444 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:36.313218    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:36.379909 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:43:36.379860    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:43:36.709448 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:36.709414    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:37.614971 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.614937    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"]
Oct 30 19:43:37.615776 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.615754    4712 cpu_manager.go:414] "containerMap: removing container" podUID="6ec82e39-a787-45d5-8584-e6d0d47ae7ff" containerName="payload-pre-processing"
Oct 30 19:43:37.615840 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.615778    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec82e39-a787-45d5-8584-e6d0d47ae7ff" containerName="payload-pre-processing"
Oct 30 19:43:37.615840 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.615792    4712 cpu_manager.go:414] "containerMap: removing container" podUID="7bef9c13-9c33-4c00-a711-d78c2c5091cb" containerName="istio-proxy"
Oct 30 19:43:37.615840 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.615798    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bef9c13-9c33-4c00-a711-d78c2c5091cb" containerName="istio-proxy"
Oct 30 19:43:37.615840 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.615826    4712 cpu_manager.go:414] "containerMap: removing container" podUID="c3dce202-6631-47f0-903d-64bb021d8011" containerName="payload-processing"
Oct 30 19:43:37.615840 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.615831    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3dce202-6631-47f0-903d-64bb021d8011" containerName="payload-processing"
Oct 30 19:43:37.615995 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.615963    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3dce202-6631-47f0-903d-64bb021d8011" containerName="payload-processing"
Oct 30 19:43:37.615995 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.615978    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec82e39-a787-45d5-8584-e6d0d47ae7ff" containerName="payload-pre-processing"
Oct 30 19:43:37.615995 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.615985    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bef9c13-9c33-4c00-a711-d78c2c5091cb" containerName="istio-proxy"
Oct 30 19:43:37.619676 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.619650    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.644206 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.637045    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-67953efd-openshift-default-dockercfg-74jjs\""
Oct 30 19:43:37.653934 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.653898    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"]
Oct 30 19:43:37.696796 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.696766    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw7m6\" (UniqueName: \"kubernetes.io/projected/700a4ae0-505b-4926-93c0-ee5b5f627030-kube-api-access-rw7m6\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.696952 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.696812    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-envoy\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.697033 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.697015    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-credential-socket\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.697089 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.697073    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-workload-socket\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.697258 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.697237    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-data\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.697338 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.697303    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700a4ae0-505b-4926-93c0-ee5b5f627030-istiod-ca-cert\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.697465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.697447    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-workload-certs\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.697530 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.697508    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-podinfo\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.697634 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.697603    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-token\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.798733 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.798705    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-workload-certs\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.799174 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.798765    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-podinfo\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.799174 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.798821    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-token\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.799174 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.798874    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw7m6\" (UniqueName: \"kubernetes.io/projected/700a4ae0-505b-4926-93c0-ee5b5f627030-kube-api-access-rw7m6\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.799174 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.798901    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-envoy\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.799174 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.799012    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-credential-socket\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.799174 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.799056    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-workload-socket\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.799174 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.799149    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-workload-certs\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.799174 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.799159    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-data\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.799597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.799242    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700a4ae0-505b-4926-93c0-ee5b5f627030-istiod-ca-cert\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.799597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.799354    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-credential-socket\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.799597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.799422    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-workload-socket\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.799709 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.799632    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-data\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.799985 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.799963    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700a4ae0-505b-4926-93c0-ee5b5f627030-istiod-ca-cert\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.801577 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.801553    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-podinfo\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.801694 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.801674    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-envoy\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.819593 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.819572    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-token\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.821068 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.821043    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw7m6\" (UniqueName: \"kubernetes.io/projected/700a4ae0-505b-4926-93c0-ee5b5f627030-kube-api-access-rw7m6\") pod \"e2e-mt-67953efd-openshift-default-f666c5445-q8lpd\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") " pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:37.948653 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:37.948632    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:38.239954 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:38.239867    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:38.284078 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:38.284048    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"]
Oct 30 19:43:38.284679 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:43:38.284653    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700a4ae0_505b_4926_93c0_ee5b5f627030.slice/crio-4335efbc25ff09add8c5e3460366e4a7c562bb6538bcc19db58a20d62e9bb8bd WatchSource:0}: Error finding container 4335efbc25ff09add8c5e3460366e4a7c562bb6538bcc19db58a20d62e9bb8bd: Status 404 returned error can't find the container with id 4335efbc25ff09add8c5e3460366e4a7c562bb6538bcc19db58a20d62e9bb8bd
Oct 30 19:43:38.287934 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:38.287898    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:43:38.288018 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:38.287965    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:43:38.288018 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:38.288013    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:43:39.081467 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:39.081426    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd" event={"ID":"700a4ae0-505b-4926-93c0-ee5b5f627030","Type":"ContainerStarted","Data":"2b83080aabe07b80575808940d4a6d0b94d856768a576eeedc903a69eed8c88f"}
Oct 30 19:43:39.081866 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:39.081508    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd" event={"ID":"700a4ae0-505b-4926-93c0-ee5b5f627030","Type":"ContainerStarted","Data":"4335efbc25ff09add8c5e3460366e4a7c562bb6538bcc19db58a20d62e9bb8bd"}
Oct 30 19:43:39.141500 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:39.141454    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd" podStartSLOduration=2.141440192 podStartE2EDuration="2.141440192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:39.137487588 +0000 UTC m=+1372.436749746" watchObservedRunningTime="2026-10-30 19:43:39.141440192 +0000 UTC m=+1372.440702321"
Oct 30 19:43:39.938866 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:39.938314    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:39.953377 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:39.952357    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:39.962410 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:39.960490    4712 patch_prober.go:29] interesting pod/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:43:39.962410 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:39.960549    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd" podUID="700a4ae0-505b-4926-93c0-ee5b5f627030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:43:40.949139 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:40.949106    4712 patch_prober.go:29] interesting pod/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:43:40.949609 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:40.949165    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd" podUID="700a4ae0-505b-4926-93c0-ee5b5f627030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:43:41.385273 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:41.385003    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:41.949787 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:41.949747    4712 patch_prober.go:29] interesting pod/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:43:41.950256 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:41.949807    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd" podUID="700a4ae0-505b-4926-93c0-ee5b5f627030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:43:42.954689 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:42.954656    4712 patch_prober.go:29] interesting pod/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:43:42.955293 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:42.955261    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd" podUID="700a4ae0-505b-4926-93c0-ee5b5f627030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:43:43.152862 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:43.152824    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:43.624761 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:43.624724    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:43.837420 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:43.831245    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:43.949531 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:43.949428    4712 patch_prober.go:29] interesting pod/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:43:43.949531 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:43.949504    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd" podUID="700a4ae0-505b-4926-93c0-ee5b5f627030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:43:44.062872 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:44.062837    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:44.449353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:44.449313    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:44.619389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:44.616643    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:44.949119 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:44.949077    4712 patch_prober.go:29] interesting pod/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 19:43:44.949292 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:44.949156    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd" podUID="700a4ae0-505b-4926-93c0-ee5b5f627030" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 19:43:45.233442 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:45.232884    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:45.953843 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:45.953812    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:45.954338 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:45.954314    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:45.954954 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:45.954937    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:43:46.938754 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:46.938722    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:47.324493 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:47.324414    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"]
Oct 30 19:43:47.324700 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:47.324674    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5" podUID="1eb94f9e-d211-4973-86f8-72b96cf14270" containerName="istio-proxy" containerID="cri-o://8f2cb52aade6b04507d0118bf08dc7e14676be7726ecab85b597f50b926a3e86" gracePeriod=30
Oct 30 19:43:47.335815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:47.335776    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"]
Oct 30 19:43:47.336008 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:47.335968    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb" podUID="028719db-ba4c-4f77-8f0f-45ed5d94e76c" containerName="istio-proxy" containerID="cri-o://f6a37b740c8ee2bfa207effcd90788d19ba5b42cf5ede883ee26d15aca9e2b72" gracePeriod=30
Oct 30 19:43:47.815162 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:47.815130    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"]
Oct 30 19:43:47.815331 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:47.815310    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl" podUID="9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" containerName="istio-proxy" containerID="cri-o://e8fc7148ef5730665cb0c40f8f37b4cc0aa2a8c2f3dc39a888655b089204ed0d" gracePeriod=30
Oct 30 19:43:48.993329 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:48.993285    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:49.364785 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:49.364702    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"]
Oct 30 19:43:49.389854 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:49.389827    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"]
Oct 30 19:43:49.616702 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:49.616673    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:50.838222 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:50.838180    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:52.963434 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:52.963406    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:43:52.992380 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:52.987959    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:43:53.084393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.083468    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-credential-socket\") pod \"1eb94f9e-d211-4973-86f8-72b96cf14270\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") "
Oct 30 19:43:53.084393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.083526    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb94f9e-d211-4973-86f8-72b96cf14270-kube-api-access-gk7bw\" (UniqueName: \"kubernetes.io/projected/1eb94f9e-d211-4973-86f8-72b96cf14270-kube-api-access-gk7bw\") pod \"1eb94f9e-d211-4973-86f8-72b96cf14270\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") "
Oct 30 19:43:53.084393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.083768    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-workload-socket\") pod \"1eb94f9e-d211-4973-86f8-72b96cf14270\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") "
Oct 30 19:43:53.084393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.083938    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-workload-certs\") pod \"1eb94f9e-d211-4973-86f8-72b96cf14270\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") "
Oct 30 19:43:53.084393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.084010    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-podinfo\") pod \"1eb94f9e-d211-4973-86f8-72b96cf14270\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") "
Oct 30 19:43:53.084393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.084075    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb94f9e-d211-4973-86f8-72b96cf14270-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb94f9e-d211-4973-86f8-72b96cf14270-istiod-ca-cert\") pod \"1eb94f9e-d211-4973-86f8-72b96cf14270\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") "
Oct 30 19:43:53.084393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.084119    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-token\" (UniqueName: \"kubernetes.io/projected/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-token\") pod \"1eb94f9e-d211-4973-86f8-72b96cf14270\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") "
Oct 30 19:43:53.084952 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.084610    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-envoy\") pod \"1eb94f9e-d211-4973-86f8-72b96cf14270\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") "
Oct 30 19:43:53.084952 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.084741    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-data\") pod \"1eb94f9e-d211-4973-86f8-72b96cf14270\" (UID: \"1eb94f9e-d211-4973-86f8-72b96cf14270\") "
Oct 30 19:43:53.122521 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.122472    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-credential-socket" pod "1eb94f9e-d211-4973-86f8-72b96cf14270" (UID: "1eb94f9e-d211-4973-86f8-72b96cf14270"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:53.123314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.123283    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb94f9e-d211-4973-86f8-72b96cf14270-istiod-ca-cert" pod "1eb94f9e-d211-4973-86f8-72b96cf14270" (UID: "1eb94f9e-d211-4973-86f8-72b96cf14270"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:53.123636 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.123612    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-workload-certs" pod "1eb94f9e-d211-4973-86f8-72b96cf14270" (UID: "1eb94f9e-d211-4973-86f8-72b96cf14270"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:53.123837 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.123816    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-workload-socket" pod "1eb94f9e-d211-4973-86f8-72b96cf14270" (UID: "1eb94f9e-d211-4973-86f8-72b96cf14270"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:53.124034 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.124013    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-data" pod "1eb94f9e-d211-4973-86f8-72b96cf14270" (UID: "1eb94f9e-d211-4973-86f8-72b96cf14270"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.190497    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-workload-certs\") pod \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") "
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.190610    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-token\" (UniqueName: \"kubernetes.io/projected/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-token\") pod \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") "
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.190667    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-envoy\") pod \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") "
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.190735    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-podinfo\") pod \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") "
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.190914    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-credential-socket\") pod \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") "
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.190995    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/028719db-ba4c-4f77-8f0f-45ed5d94e76c-kube-api-access-bcrjx\" (UniqueName: \"kubernetes.io/projected/028719db-ba4c-4f77-8f0f-45ed5d94e76c-kube-api-access-bcrjx\") pod \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") "
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.191050    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-workload-socket\") pod \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") "
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.191094    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istiod-ca-cert\") pod \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") "
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.191159    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-data\") pod \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\" (UID: \"028719db-ba4c-4f77-8f0f-45ed5d94e76c\") "
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.192005    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb94f9e-d211-4973-86f8-72b96cf14270-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.192028    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.192042    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.192054    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.192053    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-podinfo" pod "1eb94f9e-d211-4973-86f8-72b96cf14270" (UID: "1eb94f9e-d211-4973-86f8-72b96cf14270"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.192067    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.195202    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-token" pod "1eb94f9e-d211-4973-86f8-72b96cf14270" (UID: "1eb94f9e-d211-4973-86f8-72b96cf14270"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.196444    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-podinfo" pod "028719db-ba4c-4f77-8f0f-45ed5d94e76c" (UID: "028719db-ba4c-4f77-8f0f-45ed5d94e76c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.196636    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-workload-certs" pod "028719db-ba4c-4f77-8f0f-45ed5d94e76c" (UID: "028719db-ba4c-4f77-8f0f-45ed5d94e76c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.196854    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-workload-socket" pod "028719db-ba4c-4f77-8f0f-45ed5d94e76c" (UID: "028719db-ba4c-4f77-8f0f-45ed5d94e76c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.197056    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-credential-socket" pod "028719db-ba4c-4f77-8f0f-45ed5d94e76c" (UID: "028719db-ba4c-4f77-8f0f-45ed5d94e76c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.197382    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istiod-ca-cert" pod "028719db-ba4c-4f77-8f0f-45ed5d94e76c" (UID: "028719db-ba4c-4f77-8f0f-45ed5d94e76c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.197873    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-token" pod "028719db-ba4c-4f77-8f0f-45ed5d94e76c" (UID: "028719db-ba4c-4f77-8f0f-45ed5d94e76c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.199066    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028719db-ba4c-4f77-8f0f-45ed5d94e76c-kube-api-access-bcrjx" pod "028719db-ba4c-4f77-8f0f-45ed5d94e76c" (UID: "028719db-ba4c-4f77-8f0f-45ed5d94e76c"). InnerVolumeSpecName "kube-api-access-bcrjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.200183    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb94f9e-d211-4973-86f8-72b96cf14270-kube-api-access-gk7bw" pod "1eb94f9e-d211-4973-86f8-72b96cf14270" (UID: "1eb94f9e-d211-4973-86f8-72b96cf14270"). InnerVolumeSpecName "kube-api-access-gk7bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.202802    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-envoy" pod "1eb94f9e-d211-4973-86f8-72b96cf14270" (UID: "1eb94f9e-d211-4973-86f8-72b96cf14270"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:53.213435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.203153    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-envoy" pod "028719db-ba4c-4f77-8f0f-45ed5d94e76c" (UID: "028719db-ba4c-4f77-8f0f-45ed5d94e76c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:53.215437 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.215389    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-data" pod "028719db-ba4c-4f77-8f0f-45ed5d94e76c" (UID: "028719db-ba4c-4f77-8f0f-45ed5d94e76c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:53.299416 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.299335    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:53.299617 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.299597    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk7bw\" (UniqueName: \"kubernetes.io/projected/1eb94f9e-d211-4973-86f8-72b96cf14270-kube-api-access-gk7bw\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:53.299756 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.299741    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcrjx\" (UniqueName: \"kubernetes.io/projected/028719db-ba4c-4f77-8f0f-45ed5d94e76c-kube-api-access-bcrjx\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:53.299863 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.299849    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:53.299958 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.299945    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:53.300049 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.300037    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:53.300138 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.300126    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:53.300635 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.300209    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:53.301008 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.300988    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:53.301141 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.301127    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:53.301240 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.301228    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:53.301332 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.301320    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb94f9e-d211-4973-86f8-72b96cf14270-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:53.301455 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.301441    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/028719db-ba4c-4f77-8f0f-45ed5d94e76c-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:53.432967 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.432421    4712 generic.go:356] "Generic (PLEG): container finished" podID="1eb94f9e-d211-4973-86f8-72b96cf14270" containerID="8f2cb52aade6b04507d0118bf08dc7e14676be7726ecab85b597f50b926a3e86" exitCode=0
Oct 30 19:43:53.432967 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.432567    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5" event={"ID":"1eb94f9e-d211-4973-86f8-72b96cf14270","Type":"ContainerDied","Data":"8f2cb52aade6b04507d0118bf08dc7e14676be7726ecab85b597f50b926a3e86"}
Oct 30 19:43:53.432967 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.432621    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5" event={"ID":"1eb94f9e-d211-4973-86f8-72b96cf14270","Type":"ContainerDied","Data":"d97fb25084d9eeede1d060ec187e747f53d1d2fd40507932ebaaa7b859b7798d"}
Oct 30 19:43:53.432967 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.432651    4712 scope.go:122] "RemoveContainer" containerID="8f2cb52aade6b04507d0118bf08dc7e14676be7726ecab85b597f50b926a3e86"
Oct 30 19:43:53.432967 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.432825    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"
Oct 30 19:43:53.448231 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.448184    4712 generic.go:356] "Generic (PLEG): container finished" podID="9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" containerID="e8fc7148ef5730665cb0c40f8f37b4cc0aa2a8c2f3dc39a888655b089204ed0d" exitCode=0
Oct 30 19:43:53.448665 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.448641    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl" event={"ID":"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755","Type":"ContainerDied","Data":"e8fc7148ef5730665cb0c40f8f37b4cc0aa2a8c2f3dc39a888655b089204ed0d"}
Oct 30 19:43:53.451310 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.451288    4712 generic.go:356] "Generic (PLEG): container finished" podID="028719db-ba4c-4f77-8f0f-45ed5d94e76c" containerID="f6a37b740c8ee2bfa207effcd90788d19ba5b42cf5ede883ee26d15aca9e2b72" exitCode=0
Oct 30 19:43:53.451571 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.451550    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb" event={"ID":"028719db-ba4c-4f77-8f0f-45ed5d94e76c","Type":"ContainerDied","Data":"f6a37b740c8ee2bfa207effcd90788d19ba5b42cf5ede883ee26d15aca9e2b72"}
Oct 30 19:43:53.451874 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.451748    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb" event={"ID":"028719db-ba4c-4f77-8f0f-45ed5d94e76c","Type":"ContainerDied","Data":"99bfea1a71f48462fe00e6e2f7d3e772879697e2218c486e4a3fdd195107881a"}
Oct 30 19:43:53.452040 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.451851    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"
Oct 30 19:43:53.543596 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.543560    4712 scope.go:122] "RemoveContainer" containerID="8f2cb52aade6b04507d0118bf08dc7e14676be7726ecab85b597f50b926a3e86"
Oct 30 19:43:53.545243 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:43:53.545211    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f2cb52aade6b04507d0118bf08dc7e14676be7726ecab85b597f50b926a3e86\": container with ID starting with 8f2cb52aade6b04507d0118bf08dc7e14676be7726ecab85b597f50b926a3e86 not found: ID does not exist" containerID="8f2cb52aade6b04507d0118bf08dc7e14676be7726ecab85b597f50b926a3e86"
Oct 30 19:43:53.545497 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.545446    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f2cb52aade6b04507d0118bf08dc7e14676be7726ecab85b597f50b926a3e86"} err="failed to get container status \"8f2cb52aade6b04507d0118bf08dc7e14676be7726ecab85b597f50b926a3e86\": rpc error: code = NotFound desc = could not find container \"8f2cb52aade6b04507d0118bf08dc7e14676be7726ecab85b597f50b926a3e86\": container with ID starting with 8f2cb52aade6b04507d0118bf08dc7e14676be7726ecab85b597f50b926a3e86 not found: ID does not exist"
Oct 30 19:43:53.545638 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.545622    4712 scope.go:122] "RemoveContainer" containerID="f6a37b740c8ee2bfa207effcd90788d19ba5b42cf5ede883ee26d15aca9e2b72"
Oct 30 19:43:53.800610 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.760583    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:43:53.866168 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.866135    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"]
Oct 30 19:43:53.903039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.894554    4712 scope.go:122] "RemoveContainer" containerID="f6a37b740c8ee2bfa207effcd90788d19ba5b42cf5ede883ee26d15aca9e2b72"
Oct 30 19:43:53.903039 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:43:53.894952    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a37b740c8ee2bfa207effcd90788d19ba5b42cf5ede883ee26d15aca9e2b72\": container with ID starting with f6a37b740c8ee2bfa207effcd90788d19ba5b42cf5ede883ee26d15aca9e2b72 not found: ID does not exist" containerID="f6a37b740c8ee2bfa207effcd90788d19ba5b42cf5ede883ee26d15aca9e2b72"
Oct 30 19:43:53.903039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.894996    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a37b740c8ee2bfa207effcd90788d19ba5b42cf5ede883ee26d15aca9e2b72"} err="failed to get container status \"f6a37b740c8ee2bfa207effcd90788d19ba5b42cf5ede883ee26d15aca9e2b72\": rpc error: code = NotFound desc = could not find container \"f6a37b740c8ee2bfa207effcd90788d19ba5b42cf5ede883ee26d15aca9e2b72\": container with ID starting with f6a37b740c8ee2bfa207effcd90788d19ba5b42cf5ede883ee26d15aca9e2b72 not found: ID does not exist"
Oct 30 19:43:53.903039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.901206    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-podinfo\") pod \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") "
Oct 30 19:43:53.903039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.901392    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-workload-certs\") pod \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") "
Oct 30 19:43:53.903039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.901720    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-credential-socket\") pod \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") "
Oct 30 19:43:53.903039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.902017    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-token\" (UniqueName: \"kubernetes.io/projected/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-token\") pod \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") "
Oct 30 19:43:53.903039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.902166    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-data\") pod \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") "
Oct 30 19:43:53.903039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.902254    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istiod-ca-cert\") pod \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") "
Oct 30 19:43:53.903039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.902315    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-workload-socket\") pod \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") "
Oct 30 19:43:53.903039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.902502    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-envoy\") pod \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") "
Oct 30 19:43:53.903039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.902540    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-kube-api-access-dr4ss\" (UniqueName: \"kubernetes.io/projected/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-kube-api-access-dr4ss\") pod \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\" (UID: \"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755\") "
Oct 30 19:43:53.903825 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.903795    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-workload-socket" pod "9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" (UID: "9d0a8af7-9b2d-4a53-8be6-703ed9ab2755"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:53.904077 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.904047    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-data" pod "9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" (UID: "9d0a8af7-9b2d-4a53-8be6-703ed9ab2755"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:53.962584 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.957424    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-token" pod "9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" (UID: "9d0a8af7-9b2d-4a53-8be6-703ed9ab2755"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:53.990486 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.988009    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-credential-socket" pod "9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" (UID: "9d0a8af7-9b2d-4a53-8be6-703ed9ab2755"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:53.990486 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.989501    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istiod-ca-cert" pod "9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" (UID: "9d0a8af7-9b2d-4a53-8be6-703ed9ab2755"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:53.991307 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.991268    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-podinfo" pod "9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" (UID: "9d0a8af7-9b2d-4a53-8be6-703ed9ab2755"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:43:53.994007 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:53.993970    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-kube-api-access-dr4ss" pod "9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" (UID: "9d0a8af7-9b2d-4a53-8be6-703ed9ab2755"). InnerVolumeSpecName "kube-api-access-dr4ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:54.090984 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.007908    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:54.091233 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.091207    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:54.091383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.091352    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:54.091485 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.091473    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:54.091596 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.091583    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:54.091720 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.091688    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr4ss\" (UniqueName: \"kubernetes.io/projected/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-kube-api-access-dr4ss\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:54.123338 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.091809    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:54.123577 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.045310    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-workload-certs" pod "9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" (UID: "9d0a8af7-9b2d-4a53-8be6-703ed9ab2755"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:54.146429 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.127538    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-s7zt5"]
Oct 30 19:43:54.278388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.273530    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-envoy" pod "9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" (UID: "9d0a8af7-9b2d-4a53-8be6-703ed9ab2755"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:54.296433 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.279964    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:54.296646 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.296627    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:43:54.296741 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.283542    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"]
Oct 30 19:43:54.296864 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.296852    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-dr8wb"]
Oct 30 19:43:54.343430 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.342621    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:54.392384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.391968    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:54.398384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.393202    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:43:54.425465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.425429    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:43:54.445468 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.445431    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:54.527593 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.527542    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:54.568406 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.562745    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:54.568406 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.563159    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl" event={"ID":"9d0a8af7-9b2d-4a53-8be6-703ed9ab2755","Type":"ContainerDied","Data":"9434c95f66e3a01798f4101a24f7e726908e4f47a1e3d83c1a0ebddafbdd6460"}
Oct 30 19:43:54.568406 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.563219    4712 scope.go:122] "RemoveContainer" containerID="e8fc7148ef5730665cb0c40f8f37b4cc0aa2a8c2f3dc39a888655b089204ed0d"
Oct 30 19:43:54.568406 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.563322    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"
Oct 30 19:43:54.773385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.772786    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"]
Oct 30 19:43:54.845386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.844632    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-4csbl"]
Oct 30 19:43:54.891385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.890851    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:54.989438 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.989047    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="028719db-ba4c-4f77-8f0f-45ed5d94e76c" path="/var/lib/kubelet/pods/028719db-ba4c-4f77-8f0f-45ed5d94e76c/volumes"
Oct 30 19:43:55.000576 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.998948    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:55.000576 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:54.999999    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb94f9e-d211-4973-86f8-72b96cf14270" path="/var/lib/kubelet/pods/1eb94f9e-d211-4973-86f8-72b96cf14270/volumes"
Oct 30 19:43:55.001091 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.000766    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" path="/var/lib/kubelet/pods/9d0a8af7-9b2d-4a53-8be6-703ed9ab2755/volumes"
Oct 30 19:43:55.052384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.051672    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:55.052384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.052010    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:55.052384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.052280    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 19:43:55.060383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.052752    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 19:43:55.115383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.114534    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:55.319444 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.316539    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:43:55.319444 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.316913    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:55.387387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.386474    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:55.387387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.386890    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:43:55.397381 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.394434    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:55.397381 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.394754    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:43:55.397381 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.395059    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:43:55.397381 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.395112    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:43:55.436057 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.434887    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:55.438800 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.436454    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:43:55.455796 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.455764    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:55.499388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.476795    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:43:55.564386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.528447    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc" podUID="fd766cbc-5850-4181-95fa-a494ed7c8736" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:55.564386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.529130    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:43:55.711948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.704997    4712 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:55.720384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.719502    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:43:55.732380 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.728330    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc"
Oct 30 19:43:55.813384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.770114    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:55.824391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.823973    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:43:55.858382 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.851179    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:43:55.858382 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.852747    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:43:55.870825 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:55.865356    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:43:56.180654 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:56.180618    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:56.976783 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:56.976620    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:57.145608 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:57.145581    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:57.452480 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:57.452444    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:58.128455 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:58.128117    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:43:59.692741 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:43:59.692677    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:01.692157 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:01.692121    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:01.846441 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:01.846411    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:01.928502 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:01.928469    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"]
Oct 30 19:44:01.928690 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:01.928657    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr" podUID="15c32bf5-e05e-41bd-8482-22325ad0bf0c" containerName="istio-proxy" containerID="cri-o://27e62fca93e5cfdc95a6f7124756da1b984ce0803060b2ce6a06400da6451c32" gracePeriod=30
Oct 30 19:44:01.939653 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:01.939626    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"]
Oct 30 19:44:01.939846 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:01.939825    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk" podUID="7f14a509-cafc-4031-ba92-be8d6b919835" containerName="istio-proxy" containerID="cri-o://ff96eb35dddd08c7c6176c419e872059818ff880b002a0e6f3b888af94187218" gracePeriod=30
Oct 30 19:44:02.696623 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:02.696572    4712 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:44:02.697195 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:02.696679    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh" podUID="b4ea703f-3d59-4204-9630-e5915516c074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 19:44:03.124466 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:03.116157    4712 generic.go:356] "Generic (PLEG): container finished" podID="b4ea703f-3d59-4204-9630-e5915516c074" containerID="7abee6196605e7ab09e2cc31da25c149f0da1578809c563848315a264d9b0ed6" exitCode=0
Oct 30 19:44:03.124466 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:03.116198    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh" event={"ID":"b4ea703f-3d59-4204-9630-e5915516c074","Type":"ContainerDied","Data":"7abee6196605e7ab09e2cc31da25c149f0da1578809c563848315a264d9b0ed6"}
Oct 30 19:44:03.124466 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:03.116744    4712 scope.go:122] "RemoveContainer" containerID="7abee6196605e7ab09e2cc31da25c149f0da1578809c563848315a264d9b0ed6"
Oct 30 19:44:03.124466 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:03.118977    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:44:03.124466 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:03.119070    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:44:03.124466 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:03.119116    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:44:03.654955 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:03.654919    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:04.139170 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.139138    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh" event={"ID":"b4ea703f-3d59-4204-9630-e5915516c074","Type":"ContainerStarted","Data":"4719c1e72b41ea6049d29de1aef234688f687bd62d0a11c1f6fead039f9d329c"}
Oct 30 19:44:04.386570 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.386534    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main" containerID="cri-o://7e72ca4902d11d9ded49d237498d67736afc4e50b614b93632eebd49dbada287" gracePeriod=45
Oct 30 19:44:04.429813 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.429623    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main" containerID="cri-o://9684c490762d9b811eb286d3a81102c7278a859d8edadd2753caa612176dddef" gracePeriod=45
Oct 30 19:44:04.851785 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.848666    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:44:04.862912 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.854303    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:44:04.874583 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.874428    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:04.967888 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.967799    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-dshm\") pod \"bbf52b02-27b0-485c-9e4d-9a804a8853db\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") "
Oct 30 19:44:04.967888 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.967870    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0615cd7-08e0-4930-937c-8027e74e57e2-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0615cd7-08e0-4930-937c-8027e74e57e2-tls-certs\") pod \"b0615cd7-08e0-4930-937c-8027e74e57e2\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") "
Oct 30 19:44:04.968112 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.967981    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-dshm\") pod \"b0615cd7-08e0-4930-937c-8027e74e57e2\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") "
Oct 30 19:44:04.968112 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.968097    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf52b02-27b0-485c-9e4d-9a804a8853db-kube-api-access-6d2s9\" (UniqueName: \"kubernetes.io/projected/bbf52b02-27b0-485c-9e4d-9a804a8853db-kube-api-access-6d2s9\") pod \"bbf52b02-27b0-485c-9e4d-9a804a8853db\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") "
Oct 30 19:44:04.968211 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.968129    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-tmp-dir\") pod \"bbf52b02-27b0-485c-9e4d-9a804a8853db\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") "
Oct 30 19:44:04.968211 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.968161    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbf52b02-27b0-485c-9e4d-9a804a8853db-tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf52b02-27b0-485c-9e4d-9a804a8853db-tls-certs\") pod \"bbf52b02-27b0-485c-9e4d-9a804a8853db\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") "
Oct 30 19:44:04.968318 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.968225    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0615cd7-08e0-4930-937c-8027e74e57e2-kube-api-access-k4qpl\" (UniqueName: \"kubernetes.io/projected/b0615cd7-08e0-4930-937c-8027e74e57e2-kube-api-access-k4qpl\") pod \"b0615cd7-08e0-4930-937c-8027e74e57e2\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") "
Oct 30 19:44:04.968318 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.968281    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-model-cache\") pod \"b0615cd7-08e0-4930-937c-8027e74e57e2\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") "
Oct 30 19:44:04.968453 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.968332    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-tmp-dir\") pod \"b0615cd7-08e0-4930-937c-8027e74e57e2\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") "
Oct 30 19:44:04.968453 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.968436    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-home\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-home\") pod \"b0615cd7-08e0-4930-937c-8027e74e57e2\" (UID: \"b0615cd7-08e0-4930-937c-8027e74e57e2\") "
Oct 30 19:44:04.968564 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.968467    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-home\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-home\") pod \"bbf52b02-27b0-485c-9e4d-9a804a8853db\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") "
Oct 30 19:44:04.968564 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.968550    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-model-cache\") pod \"bbf52b02-27b0-485c-9e4d-9a804a8853db\" (UID: \"bbf52b02-27b0-485c-9e4d-9a804a8853db\") "
Oct 30 19:44:04.969082 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.969054    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-tmp-dir" pod "bbf52b02-27b0-485c-9e4d-9a804a8853db" (UID: "bbf52b02-27b0-485c-9e4d-9a804a8853db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:04.969511 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.969483    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-model-cache" pod "bbf52b02-27b0-485c-9e4d-9a804a8853db" (UID: "bbf52b02-27b0-485c-9e4d-9a804a8853db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:04.970938 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.969763    4712 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-tmp-dir\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:04.971025 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.969973    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-tmp-dir" pod "b0615cd7-08e0-4930-937c-8027e74e57e2" (UID: "b0615cd7-08e0-4930-937c-8027e74e57e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:04.971025 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.970149    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-model-cache" pod "b0615cd7-08e0-4930-937c-8027e74e57e2" (UID: "b0615cd7-08e0-4930-937c-8027e74e57e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:04.971025 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.970353    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-home" pod "b0615cd7-08e0-4930-937c-8027e74e57e2" (UID: "b0615cd7-08e0-4930-937c-8027e74e57e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:04.971025 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.970536    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-home" pod "bbf52b02-27b0-485c-9e4d-9a804a8853db" (UID: "bbf52b02-27b0-485c-9e4d-9a804a8853db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:04.971244 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.971098    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf52b02-27b0-485c-9e4d-9a804a8853db-kube-api-access-6d2s9" pod "bbf52b02-27b0-485c-9e4d-9a804a8853db" (UID: "bbf52b02-27b0-485c-9e4d-9a804a8853db"). InnerVolumeSpecName "kube-api-access-6d2s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:04.971418 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.971301    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-dshm" pod "b0615cd7-08e0-4930-937c-8027e74e57e2" (UID: "b0615cd7-08e0-4930-937c-8027e74e57e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:04.971901 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.971867    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbf52b02-27b0-485c-9e4d-9a804a8853db-tls-certs" pod "bbf52b02-27b0-485c-9e4d-9a804a8853db" (UID: "bbf52b02-27b0-485c-9e4d-9a804a8853db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:04.973230 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.973205    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0615cd7-08e0-4930-937c-8027e74e57e2-tls-certs" pod "b0615cd7-08e0-4930-937c-8027e74e57e2" (UID: "b0615cd7-08e0-4930-937c-8027e74e57e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:04.973230 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.973216    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-dshm" pod "bbf52b02-27b0-485c-9e4d-9a804a8853db" (UID: "bbf52b02-27b0-485c-9e4d-9a804a8853db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:04.973755 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:04.973712    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0615cd7-08e0-4930-937c-8027e74e57e2-kube-api-access-k4qpl" pod "b0615cd7-08e0-4930-937c-8027e74e57e2" (UID: "b0615cd7-08e0-4930-937c-8027e74e57e2"). InnerVolumeSpecName "kube-api-access-k4qpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:05.072527 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.072494    4712 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-model-cache\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:05.072527 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.072525    4712 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-tmp-dir\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:05.072527 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.072538    4712 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-home\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:05.072774 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.072549    4712 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-home\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:05.072774 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.072561    4712 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-model-cache\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:05.072774 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.072572    4712 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf52b02-27b0-485c-9e4d-9a804a8853db-dshm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:05.072774 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.072584    4712 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0615cd7-08e0-4930-937c-8027e74e57e2-tls-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:05.072774 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.072593    4712 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0615cd7-08e0-4930-937c-8027e74e57e2-dshm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:05.072774 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.072606    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d2s9\" (UniqueName: \"kubernetes.io/projected/bbf52b02-27b0-485c-9e4d-9a804a8853db-kube-api-access-6d2s9\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:05.072774 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.072614    4712 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf52b02-27b0-485c-9e4d-9a804a8853db-tls-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:05.072774 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.072627    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4qpl\" (UniqueName: \"kubernetes.io/projected/b0615cd7-08e0-4930-937c-8027e74e57e2-kube-api-access-k4qpl\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:05.090938 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.090908    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:44:05.091116 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.091093    4712 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:44:05.091161 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.091141    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh" podUID="b4ea703f-3d59-4204-9630-e5915516c074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 19:44:05.149341 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.149307    4712 generic.go:356] "Generic (PLEG): container finished" podID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerID="9684c490762d9b811eb286d3a81102c7278a859d8edadd2753caa612176dddef" exitCode=0
Oct 30 19:44:05.149813 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.149349    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"
Oct 30 19:44:05.149813 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.149409    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" event={"ID":"b0615cd7-08e0-4930-937c-8027e74e57e2","Type":"ContainerDied","Data":"9684c490762d9b811eb286d3a81102c7278a859d8edadd2753caa612176dddef"}
Oct 30 19:44:05.149813 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.149466    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr" event={"ID":"b0615cd7-08e0-4930-937c-8027e74e57e2","Type":"ContainerDied","Data":"2f4517efe3ec1f73cfe93f5e7d1510642da03cec3c3e5bc909bbade171e4c8cb"}
Oct 30 19:44:05.149813 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.149490    4712 scope.go:122] "RemoveContainer" containerID="9684c490762d9b811eb286d3a81102c7278a859d8edadd2753caa612176dddef"
Oct 30 19:44:05.150907 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.150881    4712 generic.go:356] "Generic (PLEG): container finished" podID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerID="7e72ca4902d11d9ded49d237498d67736afc4e50b614b93632eebd49dbada287" exitCode=0
Oct 30 19:44:05.151035 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.150957    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" event={"ID":"bbf52b02-27b0-485c-9e4d-9a804a8853db","Type":"ContainerDied","Data":"7e72ca4902d11d9ded49d237498d67736afc4e50b614b93632eebd49dbada287"}
Oct 30 19:44:05.151035 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.150999    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m" event={"ID":"bbf52b02-27b0-485c-9e4d-9a804a8853db","Type":"ContainerDied","Data":"38d16b750ae07c9eafe757be5ba7dc4e352426d370351e1c8389afcf8e3753b8"}
Oct 30 19:44:05.151147 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.150972    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"
Oct 30 19:44:05.160790 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.160771    4712 scope.go:122] "RemoveContainer" containerID="9684c490762d9b811eb286d3a81102c7278a859d8edadd2753caa612176dddef"
Oct 30 19:44:05.161128 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:44:05.161029    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9684c490762d9b811eb286d3a81102c7278a859d8edadd2753caa612176dddef\": container with ID starting with 9684c490762d9b811eb286d3a81102c7278a859d8edadd2753caa612176dddef not found: ID does not exist" containerID="9684c490762d9b811eb286d3a81102c7278a859d8edadd2753caa612176dddef"
Oct 30 19:44:05.161128 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.161052    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9684c490762d9b811eb286d3a81102c7278a859d8edadd2753caa612176dddef"} err="failed to get container status \"9684c490762d9b811eb286d3a81102c7278a859d8edadd2753caa612176dddef\": rpc error: code = NotFound desc = could not find container \"9684c490762d9b811eb286d3a81102c7278a859d8edadd2753caa612176dddef\": container with ID starting with 9684c490762d9b811eb286d3a81102c7278a859d8edadd2753caa612176dddef not found: ID does not exist"
Oct 30 19:44:05.161128 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.161068    4712 scope.go:122] "RemoveContainer" containerID="7e72ca4902d11d9ded49d237498d67736afc4e50b614b93632eebd49dbada287"
Oct 30 19:44:05.170774 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.170757    4712 scope.go:122] "RemoveContainer" containerID="7e72ca4902d11d9ded49d237498d67736afc4e50b614b93632eebd49dbada287"
Oct 30 19:44:05.171032 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:44:05.171015    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e72ca4902d11d9ded49d237498d67736afc4e50b614b93632eebd49dbada287\": container with ID starting with 7e72ca4902d11d9ded49d237498d67736afc4e50b614b93632eebd49dbada287 not found: ID does not exist" containerID="7e72ca4902d11d9ded49d237498d67736afc4e50b614b93632eebd49dbada287"
Oct 30 19:44:05.171073 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.171039    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e72ca4902d11d9ded49d237498d67736afc4e50b614b93632eebd49dbada287"} err="failed to get container status \"7e72ca4902d11d9ded49d237498d67736afc4e50b614b93632eebd49dbada287\": rpc error: code = NotFound desc = could not find container \"7e72ca4902d11d9ded49d237498d67736afc4e50b614b93632eebd49dbada287\": container with ID starting with 7e72ca4902d11d9ded49d237498d67736afc4e50b614b93632eebd49dbada287 not found: ID does not exist"
Oct 30 19:44:05.203263 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.203242    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"]
Oct 30 19:44:05.243883 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.240853    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:05.260555 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.255218    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/simulator-w3-dd23d1-kserve-79c66fc974-rrpnr"]
Oct 30 19:44:05.272654 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.272509    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"]
Oct 30 19:44:05.276564 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:05.276539    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-dd23d1/distinct-w3-dd23d1-kserve-6b8d85f674-fwv4m"]
Oct 30 19:44:06.090923 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:06.090887    4712 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:44:06.091089 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:06.090940    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh" podUID="b4ea703f-3d59-4204-9630-e5915516c074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 19:44:06.375181 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:44:06.375094    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:44:06.959379 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:06.959332    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" path="/var/lib/kubelet/pods/b0615cd7-08e0-4930-937c-8027e74e57e2/volumes"
Oct 30 19:44:06.960255 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:06.960231    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" path="/var/lib/kubelet/pods/bbf52b02-27b0-485c-9e4d-9a804a8853db/volumes"
Oct 30 19:44:07.091117 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.091086    4712 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:44:07.091264 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.091139    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh" podUID="b4ea703f-3d59-4204-9630-e5915516c074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 19:44:07.265991 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.258867    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:07.265991 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.262609    4712 generic.go:356] "Generic (PLEG): container finished" podID="15c32bf5-e05e-41bd-8482-22325ad0bf0c" containerID="27e62fca93e5cfdc95a6f7124756da1b984ce0803060b2ce6a06400da6451c32" exitCode=0
Oct 30 19:44:07.265991 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.262711    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr" event={"ID":"15c32bf5-e05e-41bd-8482-22325ad0bf0c","Type":"ContainerDied","Data":"27e62fca93e5cfdc95a6f7124756da1b984ce0803060b2ce6a06400da6451c32"}
Oct 30 19:44:07.265991 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.265854    4712 generic.go:356] "Generic (PLEG): container finished" podID="7f14a509-cafc-4031-ba92-be8d6b919835" containerID="ff96eb35dddd08c7c6176c419e872059818ff880b002a0e6f3b888af94187218" exitCode=0
Oct 30 19:44:07.265991 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.265882    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk" event={"ID":"7f14a509-cafc-4031-ba92-be8d6b919835","Type":"ContainerDied","Data":"ff96eb35dddd08c7c6176c419e872059818ff880b002a0e6f3b888af94187218"}
Oct 30 19:44:07.267814 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.267793    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:44:07.332703 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.332117    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c32bf5-e05e-41bd-8482-22325ad0bf0c-kube-api-access-p8l7m\" (UniqueName: \"kubernetes.io/projected/15c32bf5-e05e-41bd-8482-22325ad0bf0c-kube-api-access-p8l7m\") pod \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") "
Oct 30 19:44:07.332703 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.332169    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-workload-socket\") pod \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") "
Oct 30 19:44:07.332703 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.332227    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istiod-ca-cert\") pod \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") "
Oct 30 19:44:07.332703 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.332408    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-workload-certs\") pod \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") "
Oct 30 19:44:07.332703 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.332501    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-credential-socket\") pod \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") "
Oct 30 19:44:07.332703 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.332581    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-data\") pod \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") "
Oct 30 19:44:07.333280 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.333252    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-envoy\") pod \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") "
Oct 30 19:44:07.333401 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.333323    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-token\" (UniqueName: \"kubernetes.io/projected/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-token\") pod \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") "
Oct 30 19:44:07.333527 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.333427    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-podinfo\") pod \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\" (UID: \"15c32bf5-e05e-41bd-8482-22325ad0bf0c\") "
Oct 30 19:44:07.336968 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.336940    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-workload-socket" pod "15c32bf5-e05e-41bd-8482-22325ad0bf0c" (UID: "15c32bf5-e05e-41bd-8482-22325ad0bf0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:07.337239 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.337218    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istiod-ca-cert" pod "15c32bf5-e05e-41bd-8482-22325ad0bf0c" (UID: "15c32bf5-e05e-41bd-8482-22325ad0bf0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:07.337479 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.337457    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-workload-certs" pod "15c32bf5-e05e-41bd-8482-22325ad0bf0c" (UID: "15c32bf5-e05e-41bd-8482-22325ad0bf0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:07.337702 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.337638    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-credential-socket" pod "15c32bf5-e05e-41bd-8482-22325ad0bf0c" (UID: "15c32bf5-e05e-41bd-8482-22325ad0bf0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:07.337919 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.337900    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-data" pod "15c32bf5-e05e-41bd-8482-22325ad0bf0c" (UID: "15c32bf5-e05e-41bd-8482-22325ad0bf0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:07.338016 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.337924    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-podinfo" pod "15c32bf5-e05e-41bd-8482-22325ad0bf0c" (UID: "15c32bf5-e05e-41bd-8482-22325ad0bf0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:44:07.340533 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.340512    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c32bf5-e05e-41bd-8482-22325ad0bf0c-kube-api-access-p8l7m" pod "15c32bf5-e05e-41bd-8482-22325ad0bf0c" (UID: "15c32bf5-e05e-41bd-8482-22325ad0bf0c"). InnerVolumeSpecName "kube-api-access-p8l7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:07.351341 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.349483    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-envoy" pod "15c32bf5-e05e-41bd-8482-22325ad0bf0c" (UID: "15c32bf5-e05e-41bd-8482-22325ad0bf0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:07.352000 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.351975    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-token" pod "15c32bf5-e05e-41bd-8482-22325ad0bf0c" (UID: "15c32bf5-e05e-41bd-8482-22325ad0bf0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:07.387738 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.387714    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:44:07.435723 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.435659    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-istio-envoy\") pod \"7f14a509-cafc-4031-ba92-be8d6b919835\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") "
Oct 30 19:44:07.435723 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.435710    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f14a509-cafc-4031-ba92-be8d6b919835-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f14a509-cafc-4031-ba92-be8d6b919835-istiod-ca-cert\") pod \"7f14a509-cafc-4031-ba92-be8d6b919835\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") "
Oct 30 19:44:07.435957 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.435738    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-credential-socket\") pod \"7f14a509-cafc-4031-ba92-be8d6b919835\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") "
Oct 30 19:44:07.435957 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.435810    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f14a509-cafc-4031-ba92-be8d6b919835-kube-api-access-945bf\" (UniqueName: \"kubernetes.io/projected/7f14a509-cafc-4031-ba92-be8d6b919835-kube-api-access-945bf\") pod \"7f14a509-cafc-4031-ba92-be8d6b919835\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") "
Oct 30 19:44:07.435957 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.435827    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f14a509-cafc-4031-ba92-be8d6b919835-istio-token\" (UniqueName: \"kubernetes.io/projected/7f14a509-cafc-4031-ba92-be8d6b919835-istio-token\") pod \"7f14a509-cafc-4031-ba92-be8d6b919835\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") "
Oct 30 19:44:07.435957 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.435848    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-workload-socket\") pod \"7f14a509-cafc-4031-ba92-be8d6b919835\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") "
Oct 30 19:44:07.435957 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.435883    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-istio-data\") pod \"7f14a509-cafc-4031-ba92-be8d6b919835\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") "
Oct 30 19:44:07.435957 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.435927    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-workload-certs\") pod \"7f14a509-cafc-4031-ba92-be8d6b919835\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") "
Oct 30 19:44:07.436320 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.435963    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f14a509-cafc-4031-ba92-be8d6b919835-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f14a509-cafc-4031-ba92-be8d6b919835-istio-podinfo\") pod \"7f14a509-cafc-4031-ba92-be8d6b919835\" (UID: \"7f14a509-cafc-4031-ba92-be8d6b919835\") "
Oct 30 19:44:07.436412 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.436395    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:07.436412 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.436412    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:07.436525 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.436420    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:07.436525 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.436429    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8l7m\" (UniqueName: \"kubernetes.io/projected/15c32bf5-e05e-41bd-8482-22325ad0bf0c-kube-api-access-p8l7m\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:07.436525 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.436439    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:07.436525 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.436448    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:07.436525 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.436455    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:07.436525 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.436463    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:07.436525 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.436470    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c32bf5-e05e-41bd-8482-22325ad0bf0c-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:07.437255 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.437215    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f14a509-cafc-4031-ba92-be8d6b919835-istiod-ca-cert" pod "7f14a509-cafc-4031-ba92-be8d6b919835" (UID: "7f14a509-cafc-4031-ba92-be8d6b919835"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:07.437743 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.437681    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-credential-socket" pod "7f14a509-cafc-4031-ba92-be8d6b919835" (UID: "7f14a509-cafc-4031-ba92-be8d6b919835"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:07.437978 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.437940    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-istio-data" pod "7f14a509-cafc-4031-ba92-be8d6b919835" (UID: "7f14a509-cafc-4031-ba92-be8d6b919835"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:07.438085 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.437972    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-workload-certs" pod "7f14a509-cafc-4031-ba92-be8d6b919835" (UID: "7f14a509-cafc-4031-ba92-be8d6b919835"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:07.438085 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.438071    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-workload-socket" pod "7f14a509-cafc-4031-ba92-be8d6b919835" (UID: "7f14a509-cafc-4031-ba92-be8d6b919835"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:07.438835 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.438807    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f14a509-cafc-4031-ba92-be8d6b919835-istio-podinfo" pod "7f14a509-cafc-4031-ba92-be8d6b919835" (UID: "7f14a509-cafc-4031-ba92-be8d6b919835"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:44:07.438938 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.438908    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f14a509-cafc-4031-ba92-be8d6b919835-kube-api-access-945bf" pod "7f14a509-cafc-4031-ba92-be8d6b919835" (UID: "7f14a509-cafc-4031-ba92-be8d6b919835"). InnerVolumeSpecName "kube-api-access-945bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:07.439470 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.439446    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-istio-envoy" pod "7f14a509-cafc-4031-ba92-be8d6b919835" (UID: "7f14a509-cafc-4031-ba92-be8d6b919835"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:07.439693 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.439674    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f14a509-cafc-4031-ba92-be8d6b919835-istio-token" pod "7f14a509-cafc-4031-ba92-be8d6b919835" (UID: "7f14a509-cafc-4031-ba92-be8d6b919835"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:07.537843 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.537753    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:07.537843 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.537791    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f14a509-cafc-4031-ba92-be8d6b919835-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:07.537843 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.537801    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:07.537843 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.537839    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f14a509-cafc-4031-ba92-be8d6b919835-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:07.537843 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.537849    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:07.538238 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.537860    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-945bf\" (UniqueName: \"kubernetes.io/projected/7f14a509-cafc-4031-ba92-be8d6b919835-kube-api-access-945bf\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:07.538238 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.537869    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f14a509-cafc-4031-ba92-be8d6b919835-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:07.538238 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.537877    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:07.538238 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:07.537886    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f14a509-cafc-4031-ba92-be8d6b919835-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:08.091578 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:08.091541    4712 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:44:08.091723 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:08.091595    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh" podUID="b4ea703f-3d59-4204-9630-e5915516c074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 19:44:08.274529 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:08.274456    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"
Oct 30 19:44:08.274685 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:08.274452    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr" event={"ID":"15c32bf5-e05e-41bd-8482-22325ad0bf0c","Type":"ContainerDied","Data":"68b9ed59cd0c4ff98a128dadda1a37ed31876b26c406edee7a160279ee114c53"}
Oct 30 19:44:08.274685 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:08.274594    4712 scope.go:122] "RemoveContainer" containerID="27e62fca93e5cfdc95a6f7124756da1b984ce0803060b2ce6a06400da6451c32"
Oct 30 19:44:08.276120 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:08.276096    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"
Oct 30 19:44:08.276249 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:08.276135    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk" event={"ID":"7f14a509-cafc-4031-ba92-be8d6b919835","Type":"ContainerDied","Data":"f153c79a4f64ddc837321ea24b9ad595d5628cc1c0699d69c264b8fcb69560bd"}
Oct 30 19:44:08.289942 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:08.289924    4712 scope.go:122] "RemoveContainer" containerID="ff96eb35dddd08c7c6176c419e872059818ff880b002a0e6f3b888af94187218"
Oct 30 19:44:08.350788 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:08.350758    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"]
Oct 30 19:44:08.361951 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:08.361927    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-69sqk"]
Oct 30 19:44:08.397433 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:08.397387    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"]
Oct 30 19:44:08.402281 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:08.402260    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-mf7tr"]
Oct 30 19:44:08.855982 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:08.855934    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c32bf5-e05e-41bd-8482-22325ad0bf0c" path="/var/lib/kubelet/pods/15c32bf5-e05e-41bd-8482-22325ad0bf0c/volumes"
Oct 30 19:44:08.856639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:08.856619    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f14a509-cafc-4031-ba92-be8d6b919835" path="/var/lib/kubelet/pods/7f14a509-cafc-4031-ba92-be8d6b919835/volumes"
Oct 30 19:44:09.091671 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:09.091635    4712 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:44:09.091838 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:09.091686    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh" podUID="b4ea703f-3d59-4204-9630-e5915516c074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 19:44:09.331385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:09.329627    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:10.095415 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:10.095348    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:44:10.095966 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:10.095950    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:44:10.096713 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:10.096691    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:44:10.711226 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:10.711186    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:10.952901 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:10.952867    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:11.237444 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:11.237396    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:11.741720 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:11.741686    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:12.001066 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:12.000974    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:12.735353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:12.735310    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:14.365713 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:14.365677    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:15.817760 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:15.817703    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:16.276942 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:16.276909    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:16.515971 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:16.515939    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:17.639489 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:17.639460    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:17.872338 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:17.872302    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:18.110617 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:18.110523    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw"]
Oct 30 19:44:18.391813 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:18.391768    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:18.578727 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:18.578689    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:20.183515 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:20.183476    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:21.822529 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:21.822497    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:23.111410 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:23.111371    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw" podUID="7d165954-8903-4093-ab3a-81488fe6be3e" containerName="payload-processing" containerID="cri-o://441137fb591c366a3f518b645716916529a8e00e214e3178cd2623290e9def95" gracePeriod=25
Oct 30 19:44:24.664110 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:24.664076    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:25.227147 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:25.227105    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:26.928005 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:26.927967    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:27.836958 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:27.836930    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:28.041745 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:28.041706    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:29.710121 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:29.710082    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:29.924884 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:29.924855    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:30.364982 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:30.364626    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw" podUID="7d165954-8903-4093-ab3a-81488fe6be3e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:44:31.016772 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:31.016692    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:32.950392 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:32.949727    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:33.179384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:33.175522    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-cdt2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:44:33.179384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:33.175607    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-cdt2t" podUID="f1c940f3-a6ba-45e5-bb26-2997810c97b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:33.179675 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:33.179646    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:33.955337 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:33.955303    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:34.293403 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:34.293293    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:34.482668 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:34.482631    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:34.672465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:34.672422    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:34.859186 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:34.859148    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:35.060628 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:35.060542    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:35.319513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:35.319411    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:35.500937 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:35.500912    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:36.373892 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:44:36.373856    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:44:36.626553 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:36.626483    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:36.799647 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:36.799618    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:36.948492 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:36.948412    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:38.620835 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:38.620804    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:39.272865 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:39.272837    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:39.855637 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:39.855604    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:40.193019 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:40.192989    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:40.363294 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:40.363262    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw" podUID="7d165954-8903-4093-ab3a-81488fe6be3e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:44:40.707549 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:40.707514    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:41.559434 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:41.559393    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:41.898385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:41.896458    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:42.924660 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:42.924626    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:43.390786 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:43.390763    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw"
Oct 30 19:44:43.418107 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:43.418080    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d165954-8903-4093-ab3a-81488fe6be3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d165954-8903-4093-ab3a-81488fe6be3e-plugins-config-volume\") pod \"7d165954-8903-4093-ab3a-81488fe6be3e\" (UID: \"7d165954-8903-4093-ab3a-81488fe6be3e\") "
Oct 30 19:44:43.418253 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:43.418233    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d165954-8903-4093-ab3a-81488fe6be3e-kube-api-access-s9wq9\" (UniqueName: \"kubernetes.io/projected/7d165954-8903-4093-ab3a-81488fe6be3e-kube-api-access-s9wq9\") pod \"7d165954-8903-4093-ab3a-81488fe6be3e\" (UID: \"7d165954-8903-4093-ab3a-81488fe6be3e\") "
Oct 30 19:44:43.418515 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:43.418491    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d165954-8903-4093-ab3a-81488fe6be3e-plugins-config-volume" pod "7d165954-8903-4093-ab3a-81488fe6be3e" (UID: "7d165954-8903-4093-ab3a-81488fe6be3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:43.419041 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:43.419023    4712 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d165954-8903-4093-ab3a-81488fe6be3e-plugins-config-volume\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:43.420224 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:43.420193    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d165954-8903-4093-ab3a-81488fe6be3e-kube-api-access-s9wq9" pod "7d165954-8903-4093-ab3a-81488fe6be3e" (UID: "7d165954-8903-4093-ab3a-81488fe6be3e"). InnerVolumeSpecName "kube-api-access-s9wq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:43.521101 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:43.521027    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9wq9\" (UniqueName: \"kubernetes.io/projected/7d165954-8903-4093-ab3a-81488fe6be3e-kube-api-access-s9wq9\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:43.712303 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:43.712266    4712 generic.go:356] "Generic (PLEG): container finished" podID="7d165954-8903-4093-ab3a-81488fe6be3e" containerID="441137fb591c366a3f518b645716916529a8e00e214e3178cd2623290e9def95" exitCode=0
Oct 30 19:44:43.712517 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:43.712312    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw"
Oct 30 19:44:43.712517 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:43.712344    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw" event={"ID":"7d165954-8903-4093-ab3a-81488fe6be3e","Type":"ContainerDied","Data":"441137fb591c366a3f518b645716916529a8e00e214e3178cd2623290e9def95"}
Oct 30 19:44:43.712517 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:43.712420    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw" event={"ID":"7d165954-8903-4093-ab3a-81488fe6be3e","Type":"ContainerDied","Data":"26c994edc5a4967b202a648a435ef520f28a1844d29588502eb3ed6b0b885d03"}
Oct 30 19:44:43.712517 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:43.712444    4712 scope.go:122] "RemoveContainer" containerID="441137fb591c366a3f518b645716916529a8e00e214e3178cd2623290e9def95"
Oct 30 19:44:43.725516 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:43.725487    4712 scope.go:122] "RemoveContainer" containerID="441137fb591c366a3f518b645716916529a8e00e214e3178cd2623290e9def95"
Oct 30 19:44:43.725862 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:44:43.725837    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"441137fb591c366a3f518b645716916529a8e00e214e3178cd2623290e9def95\": container with ID starting with 441137fb591c366a3f518b645716916529a8e00e214e3178cd2623290e9def95 not found: ID does not exist" containerID="441137fb591c366a3f518b645716916529a8e00e214e3178cd2623290e9def95"
Oct 30 19:44:43.725939 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:43.725875    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"441137fb591c366a3f518b645716916529a8e00e214e3178cd2623290e9def95"} err="failed to get container status \"441137fb591c366a3f518b645716916529a8e00e214e3178cd2623290e9def95\": rpc error: code = NotFound desc = could not find container \"441137fb591c366a3f518b645716916529a8e00e214e3178cd2623290e9def95\": container with ID starting with 441137fb591c366a3f518b645716916529a8e00e214e3178cd2623290e9def95 not found: ID does not exist"
Oct 30 19:44:43.768826 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:43.768795    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw"]
Oct 30 19:44:43.772215 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:43.772163    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-67953efd-b8797cccd-66cpw"]
Oct 30 19:44:44.787571 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:44.787539    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:44.854545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:44.854511    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d165954-8903-4093-ab3a-81488fe6be3e" path="/var/lib/kubelet/pods/7d165954-8903-4093-ab3a-81488fe6be3e/volumes"
Oct 30 19:44:45.007502 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:45.007463    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:45.086984 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:45.086909    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"]
Oct 30 19:44:45.087171 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:45.087149    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m" podUID="d218f368-4a8f-4b85-81ea-f5c545ebe5d5" containerName="istio-proxy" containerID="cri-o://de33eae9153ebe54dcba2023727c08d0bd60436aea7013a6350c8c61f900ec08" gracePeriod=30
Oct 30 19:44:45.228399 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:45.228316    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:47.512628 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:47.512592    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:47.524443 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:47.524410    4712 scope.go:122] "RemoveContainer" containerID="295975d50b68e93e21a88b176ed4b4e3eb21c3871fac55a99a325f3add6a5f58"
Oct 30 19:44:49.412871 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:49.412831    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:50.375327 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.375300    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:44:50.432943 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.432917    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-workload-socket\") pod \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") "
Oct 30 19:44:50.433274 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.432978    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istiod-ca-cert\") pod \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") "
Oct 30 19:44:50.433274 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.433038    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-token\" (UniqueName: \"kubernetes.io/projected/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-token\") pod \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") "
Oct 30 19:44:50.433274 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.433067    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-podinfo\") pod \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") "
Oct 30 19:44:50.433274 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.433184    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-workload-socket" pod "d218f368-4a8f-4b85-81ea-f5c545ebe5d5" (UID: "d218f368-4a8f-4b85-81ea-f5c545ebe5d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:50.433274 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.433218    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-workload-certs\") pod \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") "
Oct 30 19:44:50.433625 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.433307    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-kube-api-access-wg8kf\" (UniqueName: \"kubernetes.io/projected/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-kube-api-access-wg8kf\") pod \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") "
Oct 30 19:44:50.433625 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.433434    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-envoy\") pod \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") "
Oct 30 19:44:50.433625 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.433435    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istiod-ca-cert" pod "d218f368-4a8f-4b85-81ea-f5c545ebe5d5" (UID: "d218f368-4a8f-4b85-81ea-f5c545ebe5d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:44:50.433625 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.433484    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-data\") pod \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") "
Oct 30 19:44:50.433625 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.433494    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-workload-certs" pod "d218f368-4a8f-4b85-81ea-f5c545ebe5d5" (UID: "d218f368-4a8f-4b85-81ea-f5c545ebe5d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:50.433625 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.433574    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-credential-socket\") pod \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\" (UID: \"d218f368-4a8f-4b85-81ea-f5c545ebe5d5\") "
Oct 30 19:44:50.434341 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.433741    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-data" pod "d218f368-4a8f-4b85-81ea-f5c545ebe5d5" (UID: "d218f368-4a8f-4b85-81ea-f5c545ebe5d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:50.434341 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.433919    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-credential-socket" pod "d218f368-4a8f-4b85-81ea-f5c545ebe5d5" (UID: "d218f368-4a8f-4b85-81ea-f5c545ebe5d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:50.434341 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.434295    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:50.434341 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.434319    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:50.434341 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.434332    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:50.434614 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.434345    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:50.434614 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.434376    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:50.435833 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.435789    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-kube-api-access-wg8kf" pod "d218f368-4a8f-4b85-81ea-f5c545ebe5d5" (UID: "d218f368-4a8f-4b85-81ea-f5c545ebe5d5"). InnerVolumeSpecName "kube-api-access-wg8kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:50.435984 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.435835    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-envoy" pod "d218f368-4a8f-4b85-81ea-f5c545ebe5d5" (UID: "d218f368-4a8f-4b85-81ea-f5c545ebe5d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:44:50.436789 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.436766    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-token" pod "d218f368-4a8f-4b85-81ea-f5c545ebe5d5" (UID: "d218f368-4a8f-4b85-81ea-f5c545ebe5d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:50.436971 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.436947    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-podinfo" pod "d218f368-4a8f-4b85-81ea-f5c545ebe5d5" (UID: "d218f368-4a8f-4b85-81ea-f5c545ebe5d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:44:50.535611 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.535579    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:50.535611 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.535606    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:50.535611 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.535616    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg8kf\" (UniqueName: \"kubernetes.io/projected/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-kube-api-access-wg8kf\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:50.535820 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.535625    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d218f368-4a8f-4b85-81ea-f5c545ebe5d5-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:44:50.765339 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.765261    4712 generic.go:356] "Generic (PLEG): container finished" podID="d218f368-4a8f-4b85-81ea-f5c545ebe5d5" containerID="de33eae9153ebe54dcba2023727c08d0bd60436aea7013a6350c8c61f900ec08" exitCode=0
Oct 30 19:44:50.765339 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.765299    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m" event={"ID":"d218f368-4a8f-4b85-81ea-f5c545ebe5d5","Type":"ContainerDied","Data":"de33eae9153ebe54dcba2023727c08d0bd60436aea7013a6350c8c61f900ec08"}
Oct 30 19:44:50.765339 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.765312    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"
Oct 30 19:44:50.765339 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.765341    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m" event={"ID":"d218f368-4a8f-4b85-81ea-f5c545ebe5d5","Type":"ContainerDied","Data":"692f101569fd900810bd9053fe4172ecb461c7d1061c320c25cb11245c0b5447"}
Oct 30 19:44:50.765668 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.765377    4712 scope.go:122] "RemoveContainer" containerID="de33eae9153ebe54dcba2023727c08d0bd60436aea7013a6350c8c61f900ec08"
Oct 30 19:44:50.775974 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.775955    4712 scope.go:122] "RemoveContainer" containerID="9dc64d824b0197845e8b72983ba7f6ee3251fff4660f1fed2486420cce8a4e93"
Oct 30 19:44:50.785764 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.785731    4712 scope.go:122] "RemoveContainer" containerID="de33eae9153ebe54dcba2023727c08d0bd60436aea7013a6350c8c61f900ec08"
Oct 30 19:44:50.786015 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:44:50.785998    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de33eae9153ebe54dcba2023727c08d0bd60436aea7013a6350c8c61f900ec08\": container with ID starting with de33eae9153ebe54dcba2023727c08d0bd60436aea7013a6350c8c61f900ec08 not found: ID does not exist" containerID="de33eae9153ebe54dcba2023727c08d0bd60436aea7013a6350c8c61f900ec08"
Oct 30 19:44:50.786064 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.786022    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de33eae9153ebe54dcba2023727c08d0bd60436aea7013a6350c8c61f900ec08"} err="failed to get container status \"de33eae9153ebe54dcba2023727c08d0bd60436aea7013a6350c8c61f900ec08\": rpc error: code = NotFound desc = could not find container \"de33eae9153ebe54dcba2023727c08d0bd60436aea7013a6350c8c61f900ec08\": container with ID starting with de33eae9153ebe54dcba2023727c08d0bd60436aea7013a6350c8c61f900ec08 not found: ID does not exist"
Oct 30 19:44:50.786064 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.786037    4712 scope.go:122] "RemoveContainer" containerID="9dc64d824b0197845e8b72983ba7f6ee3251fff4660f1fed2486420cce8a4e93"
Oct 30 19:44:50.786249 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:44:50.786232    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc64d824b0197845e8b72983ba7f6ee3251fff4660f1fed2486420cce8a4e93\": container with ID starting with 9dc64d824b0197845e8b72983ba7f6ee3251fff4660f1fed2486420cce8a4e93 not found: ID does not exist" containerID="9dc64d824b0197845e8b72983ba7f6ee3251fff4660f1fed2486420cce8a4e93"
Oct 30 19:44:50.786292 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.786254    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc64d824b0197845e8b72983ba7f6ee3251fff4660f1fed2486420cce8a4e93"} err="failed to get container status \"9dc64d824b0197845e8b72983ba7f6ee3251fff4660f1fed2486420cce8a4e93\": rpc error: code = NotFound desc = could not find container \"9dc64d824b0197845e8b72983ba7f6ee3251fff4660f1fed2486420cce8a4e93\": container with ID starting with 9dc64d824b0197845e8b72983ba7f6ee3251fff4660f1fed2486420cce8a4e93 not found: ID does not exist"
Oct 30 19:44:50.827827 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.827802    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"]
Oct 30 19:44:50.831873 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.831852    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4a4e2f-openshift-default-57fd8d8689-2nv5m"]
Oct 30 19:44:50.853610 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:50.853586    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d218f368-4a8f-4b85-81ea-f5c545ebe5d5" path="/var/lib/kubelet/pods/d218f368-4a8f-4b85-81ea-f5c545ebe5d5/volumes"
Oct 30 19:44:51.155914 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:51.155879    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:52.686543 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:52.686504    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:54.336806 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:54.336769    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:55.836793 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:55.836745    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:56.250906 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:56.250867    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j"]
Oct 30 19:44:56.578583 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:56.578495    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:56.764633 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:56.764602    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:57.484530 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:57.484498    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:58.174349 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:58.174318    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:58.388507 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:58.388478    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:44:59.824514 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:59.824480    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"]
Oct 30 19:44:59.825108 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:59.824739    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g" podUID="96d155d9-0c83-46ed-b34a-81de99b24aa1" containerName="maas-api" containerID="cri-o://dacd04f615d5fa99a344fc29a071d618f4c3d1b2868c55308a355407252d9bd1" gracePeriod=30
Oct 30 19:44:59.975219 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:44:59.975181    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.217539    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8"]
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218534    4712 cpu_manager.go:414] "containerMap: removing container" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218554    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218570    4712 cpu_manager.go:414] "containerMap: removing container" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218579    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218592    4712 cpu_manager.go:414] "containerMap: removing container" podUID="d218f368-4a8f-4b85-81ea-f5c545ebe5d5" containerName="istio-proxy"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218600    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d218f368-4a8f-4b85-81ea-f5c545ebe5d5" containerName="istio-proxy"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218621    4712 cpu_manager.go:414] "containerMap: removing container" podUID="7f14a509-cafc-4031-ba92-be8d6b919835" containerName="istio-proxy"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218628    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f14a509-cafc-4031-ba92-be8d6b919835" containerName="istio-proxy"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218650    4712 cpu_manager.go:414] "containerMap: removing container" podUID="1eb94f9e-d211-4973-86f8-72b96cf14270" containerName="istio-proxy"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218657    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb94f9e-d211-4973-86f8-72b96cf14270" containerName="istio-proxy"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218674    4712 cpu_manager.go:414] "containerMap: removing container" podUID="15c32bf5-e05e-41bd-8482-22325ad0bf0c" containerName="istio-proxy"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218680    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c32bf5-e05e-41bd-8482-22325ad0bf0c" containerName="istio-proxy"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218693    4712 cpu_manager.go:414] "containerMap: removing container" podUID="028719db-ba4c-4f77-8f0f-45ed5d94e76c" containerName="istio-proxy"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218700    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="028719db-ba4c-4f77-8f0f-45ed5d94e76c" containerName="istio-proxy"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218713    4712 cpu_manager.go:414] "containerMap: removing container" podUID="9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" containerName="istio-proxy"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218721    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" containerName="istio-proxy"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218743    4712 cpu_manager.go:414] "containerMap: removing container" podUID="7d165954-8903-4093-ab3a-81488fe6be3e" containerName="payload-processing"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218751    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d165954-8903-4093-ab3a-81488fe6be3e" containerName="payload-processing"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218762    4712 cpu_manager.go:414] "containerMap: removing container" podUID="d218f368-4a8f-4b85-81ea-f5c545ebe5d5" containerName="istio-proxy"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.218770    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d218f368-4a8f-4b85-81ea-f5c545ebe5d5" containerName="istio-proxy"
Oct 30 19:45:00.222384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.219951    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"
Oct 30 19:45:00.237948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.223678    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f14a509-cafc-4031-ba92-be8d6b919835" containerName="istio-proxy"
Oct 30 19:45:00.237948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.223709    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb94f9e-d211-4973-86f8-72b96cf14270" containerName="istio-proxy"
Oct 30 19:45:00.237948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.223719    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c32bf5-e05e-41bd-8482-22325ad0bf0c" containerName="istio-proxy"
Oct 30 19:45:00.237948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.223731    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="028719db-ba4c-4f77-8f0f-45ed5d94e76c" containerName="istio-proxy"
Oct 30 19:45:00.237948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.223740    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0a8af7-9b2d-4a53-8be6-703ed9ab2755" containerName="istio-proxy"
Oct 30 19:45:00.237948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.223754    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d165954-8903-4093-ab3a-81488fe6be3e" containerName="payload-processing"
Oct 30 19:45:00.237948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.223763    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="d218f368-4a8f-4b85-81ea-f5c545ebe5d5" containerName="istio-proxy"
Oct 30 19:45:00.237948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.223779    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0615cd7-08e0-4930-937c-8027e74e57e2" containerName="main"
Oct 30 19:45:00.237948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.223787    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf52b02-27b0-485c-9e4d-9a804a8853db" containerName="main"
Oct 30 19:45:00.237948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.223795    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="d218f368-4a8f-4b85-81ea-f5c545ebe5d5" containerName="istio-proxy"
Oct 30 19:45:00.289383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.282451    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl"]
Oct 30 19:45:00.292521 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.291619    4712 cpu_manager.go:414] "containerMap: removing container" podUID="96d155d9-0c83-46ed-b34a-81de99b24aa1" containerName="maas-api"
Oct 30 19:45:00.292521 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.291643    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d155d9-0c83-46ed-b34a-81de99b24aa1" containerName="maas-api"
Oct 30 19:45:00.292964 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.292939    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d155d9-0c83-46ed-b34a-81de99b24aa1" containerName="maas-api"
Oct 30 19:45:00.298050 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.298021    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8"
Oct 30 19:45:00.309104 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.309070    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96d155d9-0c83-46ed-b34a-81de99b24aa1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96d155d9-0c83-46ed-b34a-81de99b24aa1-maas-api-tls\") pod \"96d155d9-0c83-46ed-b34a-81de99b24aa1\" (UID: \"96d155d9-0c83-46ed-b34a-81de99b24aa1\") "
Oct 30 19:45:00.309328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.309170    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96d155d9-0c83-46ed-b34a-81de99b24aa1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d155d9-0c83-46ed-b34a-81de99b24aa1-openshift-service-ca\") pod \"96d155d9-0c83-46ed-b34a-81de99b24aa1\" (UID: \"96d155d9-0c83-46ed-b34a-81de99b24aa1\") "
Oct 30 19:45:00.309328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.309213    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96d155d9-0c83-46ed-b34a-81de99b24aa1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96d155d9-0c83-46ed-b34a-81de99b24aa1-metrics-certs\") pod \"96d155d9-0c83-46ed-b34a-81de99b24aa1\" (UID: \"96d155d9-0c83-46ed-b34a-81de99b24aa1\") "
Oct 30 19:45:00.309328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.309321    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d155d9-0c83-46ed-b34a-81de99b24aa1-kube-api-access-vmjvx\" (UniqueName: \"kubernetes.io/projected/96d155d9-0c83-46ed-b34a-81de99b24aa1-kube-api-access-vmjvx\") pod \"96d155d9-0c83-46ed-b34a-81de99b24aa1\" (UID: \"96d155d9-0c83-46ed-b34a-81de99b24aa1\") "
Oct 30 19:45:00.310761 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.310357    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d155d9-0c83-46ed-b34a-81de99b24aa1-openshift-service-ca" pod "96d155d9-0c83-46ed-b34a-81de99b24aa1" (UID: "96d155d9-0c83-46ed-b34a-81de99b24aa1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:00.316429 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.315018    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch"]
Oct 30 19:45:00.326174 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.324951    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d155d9-0c83-46ed-b34a-81de99b24aa1-metrics-certs" pod "96d155d9-0c83-46ed-b34a-81de99b24aa1" (UID: "96d155d9-0c83-46ed-b34a-81de99b24aa1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:00.330147 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.327619    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae23e2d-5493-45ce-b64a-42e8bc7badcf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8\" (UID: \"4ae23e2d-5493-45ce-b64a-42e8bc7badcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8"
Oct 30 19:45:00.330147 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.327721    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d155d9-0c83-46ed-b34a-81de99b24aa1-maas-api-tls" pod "96d155d9-0c83-46ed-b34a-81de99b24aa1" (UID: "96d155d9-0c83-46ed-b34a-81de99b24aa1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:00.330147 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.327817    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlhc9\" (UniqueName: \"kubernetes.io/projected/4ae23e2d-5493-45ce-b64a-42e8bc7badcf-kube-api-access-hlhc9\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8\" (UID: \"4ae23e2d-5493-45ce-b64a-42e8bc7badcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8"
Oct 30 19:45:00.330147 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.327965    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96d155d9-0c83-46ed-b34a-81de99b24aa1-maas-api-tls\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:00.330147 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.327982    4712 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d155d9-0c83-46ed-b34a-81de99b24aa1-openshift-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:00.330147 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.327997    4712 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96d155d9-0c83-46ed-b34a-81de99b24aa1-metrics-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:00.331203 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.330562    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl"
Oct 30 19:45:00.331872 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.331807    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch"
Oct 30 19:45:00.339070 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.337416    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8"]
Oct 30 19:45:00.364156 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.357450    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl"]
Oct 30 19:45:00.380255 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.380224    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4qb5\""
Oct 30 19:45:00.384586 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.384496    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch"]
Oct 30 19:45:00.403838 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.403770    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r"]
Oct 30 19:45:00.422426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.422200    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d155d9-0c83-46ed-b34a-81de99b24aa1-kube-api-access-vmjvx" pod "96d155d9-0c83-46ed-b34a-81de99b24aa1" (UID: "96d155d9-0c83-46ed-b34a-81de99b24aa1"). InnerVolumeSpecName "kube-api-access-vmjvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:00.425539 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.425512    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r"]
Oct 30 19:45:00.425754 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.425737    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r"
Oct 30 19:45:00.435211 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.435176    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7126065-03f2-422f-ace7-566b947ed91a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl\" (UID: \"e7126065-03f2-422f-ace7-566b947ed91a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl"
Oct 30 19:45:00.435413 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.435351    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6a1252-2465-41d9-9a09-96489fd30f66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch\" (UID: \"0c6a1252-2465-41d9-9a09-96489fd30f66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch"
Oct 30 19:45:00.435673 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.435651    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzbw9\" (UniqueName: \"kubernetes.io/projected/0c6a1252-2465-41d9-9a09-96489fd30f66-kube-api-access-mzbw9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch\" (UID: \"0c6a1252-2465-41d9-9a09-96489fd30f66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch"
Oct 30 19:45:00.435956 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.435922    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae23e2d-5493-45ce-b64a-42e8bc7badcf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8\" (UID: \"4ae23e2d-5493-45ce-b64a-42e8bc7badcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8"
Oct 30 19:45:00.436102 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.436027    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t55fm\" (UniqueName: \"kubernetes.io/projected/e7126065-03f2-422f-ace7-566b947ed91a-kube-api-access-t55fm\") pod \"maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl\" (UID: \"e7126065-03f2-422f-ace7-566b947ed91a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl"
Oct 30 19:45:00.436440 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.436419    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlhc9\" (UniqueName: \"kubernetes.io/projected/4ae23e2d-5493-45ce-b64a-42e8bc7badcf-kube-api-access-hlhc9\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8\" (UID: \"4ae23e2d-5493-45ce-b64a-42e8bc7badcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8"
Oct 30 19:45:00.436758 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.436739    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmjvx\" (UniqueName: \"kubernetes.io/projected/96d155d9-0c83-46ed-b34a-81de99b24aa1-kube-api-access-vmjvx\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:00.438447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.438421    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae23e2d-5493-45ce-b64a-42e8bc7badcf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8\" (UID: \"4ae23e2d-5493-45ce-b64a-42e8bc7badcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8"
Oct 30 19:45:00.463938 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.463915    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlhc9\" (UniqueName: \"kubernetes.io/projected/4ae23e2d-5493-45ce-b64a-42e8bc7badcf-kube-api-access-hlhc9\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8\" (UID: \"4ae23e2d-5493-45ce-b64a-42e8bc7badcf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8"
Oct 30 19:45:00.538198 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.538053    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtgdc\" (UniqueName: \"kubernetes.io/projected/5c79a999-80c1-44fa-b15b-2230defbbbd2-kube-api-access-wtgdc\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r\" (UID: \"5c79a999-80c1-44fa-b15b-2230defbbbd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r"
Oct 30 19:45:00.538198 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.538186    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c79a999-80c1-44fa-b15b-2230defbbbd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r\" (UID: \"5c79a999-80c1-44fa-b15b-2230defbbbd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r"
Oct 30 19:45:00.538438 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.538272    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7126065-03f2-422f-ace7-566b947ed91a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl\" (UID: \"e7126065-03f2-422f-ace7-566b947ed91a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl"
Oct 30 19:45:00.538438 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.538356    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6a1252-2465-41d9-9a09-96489fd30f66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch\" (UID: \"0c6a1252-2465-41d9-9a09-96489fd30f66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch"
Oct 30 19:45:00.538514 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.538445    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzbw9\" (UniqueName: \"kubernetes.io/projected/0c6a1252-2465-41d9-9a09-96489fd30f66-kube-api-access-mzbw9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch\" (UID: \"0c6a1252-2465-41d9-9a09-96489fd30f66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch"
Oct 30 19:45:00.538514 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.538489    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t55fm\" (UniqueName: \"kubernetes.io/projected/e7126065-03f2-422f-ace7-566b947ed91a-kube-api-access-t55fm\") pod \"maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl\" (UID: \"e7126065-03f2-422f-ace7-566b947ed91a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl"
Oct 30 19:45:00.539268 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.539249    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7126065-03f2-422f-ace7-566b947ed91a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl\" (UID: \"e7126065-03f2-422f-ace7-566b947ed91a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl"
Oct 30 19:45:00.539830 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.539805    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6a1252-2465-41d9-9a09-96489fd30f66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch\" (UID: \"0c6a1252-2465-41d9-9a09-96489fd30f66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch"
Oct 30 19:45:00.574477 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.574443    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzbw9\" (UniqueName: \"kubernetes.io/projected/0c6a1252-2465-41d9-9a09-96489fd30f66-kube-api-access-mzbw9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch\" (UID: \"0c6a1252-2465-41d9-9a09-96489fd30f66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch"
Oct 30 19:45:00.577080 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.577047    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t55fm\" (UniqueName: \"kubernetes.io/projected/e7126065-03f2-422f-ace7-566b947ed91a-kube-api-access-t55fm\") pod \"maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl\" (UID: \"e7126065-03f2-422f-ace7-566b947ed91a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl"
Oct 30 19:45:00.639813 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.639783    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtgdc\" (UniqueName: \"kubernetes.io/projected/5c79a999-80c1-44fa-b15b-2230defbbbd2-kube-api-access-wtgdc\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r\" (UID: \"5c79a999-80c1-44fa-b15b-2230defbbbd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r"
Oct 30 19:45:00.639994 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.639947    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c79a999-80c1-44fa-b15b-2230defbbbd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r\" (UID: \"5c79a999-80c1-44fa-b15b-2230defbbbd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r"
Oct 30 19:45:00.640719 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.640697    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c79a999-80c1-44fa-b15b-2230defbbbd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r\" (UID: \"5c79a999-80c1-44fa-b15b-2230defbbbd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r"
Oct 30 19:45:00.660557 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.660539    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtgdc\" (UniqueName: \"kubernetes.io/projected/5c79a999-80c1-44fa-b15b-2230defbbbd2-kube-api-access-wtgdc\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r\" (UID: \"5c79a999-80c1-44fa-b15b-2230defbbbd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r"
Oct 30 19:45:00.696566 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.696546    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8"
Oct 30 19:45:00.727370 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.727332    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl"
Oct 30 19:45:00.749292 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.749252    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch"
Oct 30 19:45:00.754109 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.754074    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r"
Oct 30 19:45:00.881508 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.880333    4712 generic.go:356] "Generic (PLEG): container finished" podID="96d155d9-0c83-46ed-b34a-81de99b24aa1" containerID="dacd04f615d5fa99a344fc29a071d618f4c3d1b2868c55308a355407252d9bd1" exitCode=0
Oct 30 19:45:00.881508 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.880506    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"
Oct 30 19:45:00.881508 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.880535    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g" event={"ID":"96d155d9-0c83-46ed-b34a-81de99b24aa1","Type":"ContainerDied","Data":"dacd04f615d5fa99a344fc29a071d618f4c3d1b2868c55308a355407252d9bd1"}
Oct 30 19:45:00.881508 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.880581    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g" event={"ID":"96d155d9-0c83-46ed-b34a-81de99b24aa1","Type":"ContainerDied","Data":"f8020a4eca79dbf4663aeccb53c4b43e7a37a9d5c3b2202d6dbc4df061ad2b17"}
Oct 30 19:45:00.881508 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.880611    4712 scope.go:122] "RemoveContainer" containerID="dacd04f615d5fa99a344fc29a071d618f4c3d1b2868c55308a355407252d9bd1"
Oct 30 19:45:00.897971 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.897939    4712 scope.go:122] "RemoveContainer" containerID="dacd04f615d5fa99a344fc29a071d618f4c3d1b2868c55308a355407252d9bd1"
Oct 30 19:45:00.902188 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:00.902159    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dacd04f615d5fa99a344fc29a071d618f4c3d1b2868c55308a355407252d9bd1\": container with ID starting with dacd04f615d5fa99a344fc29a071d618f4c3d1b2868c55308a355407252d9bd1 not found: ID does not exist" containerID="dacd04f615d5fa99a344fc29a071d618f4c3d1b2868c55308a355407252d9bd1"
Oct 30 19:45:00.902282 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.902220    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dacd04f615d5fa99a344fc29a071d618f4c3d1b2868c55308a355407252d9bd1"} err="failed to get container status \"dacd04f615d5fa99a344fc29a071d618f4c3d1b2868c55308a355407252d9bd1\": rpc error: code = NotFound desc = could not find container \"dacd04f615d5fa99a344fc29a071d618f4c3d1b2868c55308a355407252d9bd1\": container with ID starting with dacd04f615d5fa99a344fc29a071d618f4c3d1b2868c55308a355407252d9bd1 not found: ID does not exist"
Oct 30 19:45:00.975602 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.975562    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"]
Oct 30 19:45:00.980767 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:00.980742    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd23d1-7c495f5bc8-62n8g"]
Oct 30 19:45:01.041241 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.041162    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8"]
Oct 30 19:45:01.041865 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:45:01.041837    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae23e2d_5493_45ce_b64a_42e8bc7badcf.slice/crio-29b6e48334a34be5dda5f98cf049aa3c8cd9278142a4b83387bf5967e62354a6 WatchSource:0}: Error finding container 29b6e48334a34be5dda5f98cf049aa3c8cd9278142a4b83387bf5967e62354a6: Status 404 returned error can't find the container with id 29b6e48334a34be5dda5f98cf049aa3c8cd9278142a4b83387bf5967e62354a6
Oct 30 19:45:01.045075 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.045056    4712 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:45:01.095513 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:45:01.095481    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7126065_03f2_422f_ace7_566b947ed91a.slice/crio-b2a5a98e343b23cfcf3b7a42e74242aae3f034509158077bd19358f6e8413114 WatchSource:0}: Error finding container b2a5a98e343b23cfcf3b7a42e74242aae3f034509158077bd19358f6e8413114: Status 404 returned error can't find the container with id b2a5a98e343b23cfcf3b7a42e74242aae3f034509158077bd19358f6e8413114
Oct 30 19:45:01.096086 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.096064    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl"]
Oct 30 19:45:01.147203 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.147140    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:01.154701 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:45:01.154676    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6a1252_2465_41d9_9a09_96489fd30f66.slice/crio-6b9c92efb4464a891438ee8aa8312d42ea60f5690a08b7f6ad9f000fe47a0633 WatchSource:0}: Error finding container 6b9c92efb4464a891438ee8aa8312d42ea60f5690a08b7f6ad9f000fe47a0633: Status 404 returned error can't find the container with id 6b9c92efb4464a891438ee8aa8312d42ea60f5690a08b7f6ad9f000fe47a0633
Oct 30 19:45:01.156131 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.156106    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch"]
Oct 30 19:45:01.158240 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:45:01.158220    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c79a999_80c1_44fa_b15b_2230defbbbd2.slice/crio-5f3e52a06de4353344cf87f76d24b0fc71c2fabd7661b599c27024d496df55b2 WatchSource:0}: Error finding container 5f3e52a06de4353344cf87f76d24b0fc71c2fabd7661b599c27024d496df55b2: Status 404 returned error can't find the container with id 5f3e52a06de4353344cf87f76d24b0fc71c2fabd7661b599c27024d496df55b2
Oct 30 19:45:01.161547 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.161484    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r"]
Oct 30 19:45:01.251456 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.251400    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j" podUID="abb8e01c-a453-450b-a7bb-260879385140" containerName="payload-processing" containerID="cri-o://3769fdea6c8d3a83ba3d1d7dcf0c7c38b00325ae84e4ad65bac9668aed5963d5" gracePeriod=25
Oct 30 19:45:01.344016 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.343988    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:01.513509 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.513486    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j"
Oct 30 19:45:01.557177 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.557139    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abb8e01c-a453-450b-a7bb-260879385140-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb8e01c-a453-450b-a7bb-260879385140-plugins-config-volume\") pod \"abb8e01c-a453-450b-a7bb-260879385140\" (UID: \"abb8e01c-a453-450b-a7bb-260879385140\") "
Oct 30 19:45:01.557311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.557200    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb8e01c-a453-450b-a7bb-260879385140-kube-api-access-hn5xr\" (UniqueName: \"kubernetes.io/projected/abb8e01c-a453-450b-a7bb-260879385140-kube-api-access-hn5xr\") pod \"abb8e01c-a453-450b-a7bb-260879385140\" (UID: \"abb8e01c-a453-450b-a7bb-260879385140\") "
Oct 30 19:45:01.557586 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.557559    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb8e01c-a453-450b-a7bb-260879385140-plugins-config-volume" pod "abb8e01c-a453-450b-a7bb-260879385140" (UID: "abb8e01c-a453-450b-a7bb-260879385140"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:01.557923 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.557905    4712 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb8e01c-a453-450b-a7bb-260879385140-plugins-config-volume\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:01.559208 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.559187    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb8e01c-a453-450b-a7bb-260879385140-kube-api-access-hn5xr" pod "abb8e01c-a453-450b-a7bb-260879385140" (UID: "abb8e01c-a453-450b-a7bb-260879385140"). InnerVolumeSpecName "kube-api-access-hn5xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:01.658963 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.658930    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn5xr\" (UniqueName: \"kubernetes.io/projected/abb8e01c-a453-450b-a7bb-260879385140-kube-api-access-hn5xr\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:01.701569 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.701493    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:01.893676 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.893596    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r" event={"ID":"5c79a999-80c1-44fa-b15b-2230defbbbd2","Type":"ContainerStarted","Data":"5f3e52a06de4353344cf87f76d24b0fc71c2fabd7661b599c27024d496df55b2"}
Oct 30 19:45:01.897270 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.897241    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch" event={"ID":"0c6a1252-2465-41d9-9a09-96489fd30f66","Type":"ContainerStarted","Data":"6b9c92efb4464a891438ee8aa8312d42ea60f5690a08b7f6ad9f000fe47a0633"}
Oct 30 19:45:01.899330 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.899290    4712 generic.go:356] "Generic (PLEG): container finished" podID="abb8e01c-a453-450b-a7bb-260879385140" containerID="3769fdea6c8d3a83ba3d1d7dcf0c7c38b00325ae84e4ad65bac9668aed5963d5" exitCode=0
Oct 30 19:45:01.899487 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.899338    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j"
Oct 30 19:45:01.899542 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.899472    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j" event={"ID":"abb8e01c-a453-450b-a7bb-260879385140","Type":"ContainerDied","Data":"3769fdea6c8d3a83ba3d1d7dcf0c7c38b00325ae84e4ad65bac9668aed5963d5"}
Oct 30 19:45:01.899994 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.899543    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j" event={"ID":"abb8e01c-a453-450b-a7bb-260879385140","Type":"ContainerDied","Data":"e78cf3903411adce1ad8130746aa0cc9e0d3b867adf394689855b4221bfc1aae"}
Oct 30 19:45:01.900097 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.899735    4712 scope.go:122] "RemoveContainer" containerID="3769fdea6c8d3a83ba3d1d7dcf0c7c38b00325ae84e4ad65bac9668aed5963d5"
Oct 30 19:45:01.911238 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.910233    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl" event={"ID":"e7126065-03f2-422f-ace7-566b947ed91a","Type":"ContainerStarted","Data":"b2a5a98e343b23cfcf3b7a42e74242aae3f034509158077bd19358f6e8413114"}
Oct 30 19:45:01.913326 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.913298    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8" event={"ID":"4ae23e2d-5493-45ce-b64a-42e8bc7badcf","Type":"ContainerStarted","Data":"29b6e48334a34be5dda5f98cf049aa3c8cd9278142a4b83387bf5967e62354a6"}
Oct 30 19:45:01.922956 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.922938    4712 scope.go:122] "RemoveContainer" containerID="3769fdea6c8d3a83ba3d1d7dcf0c7c38b00325ae84e4ad65bac9668aed5963d5"
Oct 30 19:45:01.923389 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:01.923345    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3769fdea6c8d3a83ba3d1d7dcf0c7c38b00325ae84e4ad65bac9668aed5963d5\": container with ID starting with 3769fdea6c8d3a83ba3d1d7dcf0c7c38b00325ae84e4ad65bac9668aed5963d5 not found: ID does not exist" containerID="3769fdea6c8d3a83ba3d1d7dcf0c7c38b00325ae84e4ad65bac9668aed5963d5"
Oct 30 19:45:01.923500 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.923401    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3769fdea6c8d3a83ba3d1d7dcf0c7c38b00325ae84e4ad65bac9668aed5963d5"} err="failed to get container status \"3769fdea6c8d3a83ba3d1d7dcf0c7c38b00325ae84e4ad65bac9668aed5963d5\": rpc error: code = NotFound desc = could not find container \"3769fdea6c8d3a83ba3d1d7dcf0c7c38b00325ae84e4ad65bac9668aed5963d5\": container with ID starting with 3769fdea6c8d3a83ba3d1d7dcf0c7c38b00325ae84e4ad65bac9668aed5963d5 not found: ID does not exist"
Oct 30 19:45:01.962713 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.962328    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j"]
Oct 30 19:45:01.973555 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:01.973525    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-086f2a-7884d5b9c4-wvx9j"]
Oct 30 19:45:02.715964 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:02.715908    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:02.874861 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:02.874715    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96d155d9-0c83-46ed-b34a-81de99b24aa1" path="/var/lib/kubelet/pods/96d155d9-0c83-46ed-b34a-81de99b24aa1/volumes"
Oct 30 19:45:02.875527 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:02.875500    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abb8e01c-a453-450b-a7bb-260879385140" path="/var/lib/kubelet/pods/abb8e01c-a453-450b-a7bb-260879385140/volumes"
Oct 30 19:45:02.996998 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:02.996901    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:04.911562 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:04.900925    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:05.024754 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:05.024644    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r" podStartSLOduration=1.751161814 podStartE2EDuration="5.024625782s" totalImagesPullingTime="3.273463968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:05.019682469 +0000 UTC m=+1458.318944601" watchObservedRunningTime="2026-10-30 19:45:05.024625782 +0000 UTC m=+1458.323887917"
Oct 30 19:45:05.966497 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:05.966457    4712 generic.go:356] "Generic (PLEG): container finished" podID="0c6a1252-2465-41d9-9a09-96489fd30f66" containerID="576d9492a031c07438bee615fd16cd948d5122305a3c2dbf2f6b7558f1c69372" exitCode=60
Oct 30 19:45:05.966961 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:05.966540    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch" event={"ID":"0c6a1252-2465-41d9-9a09-96489fd30f66","Type":"ContainerDied","Data":"576d9492a031c07438bee615fd16cd948d5122305a3c2dbf2f6b7558f1c69372"}
Oct 30 19:45:05.966961 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:05.966874    4712 scope.go:122] "RemoveContainer" containerID="576d9492a031c07438bee615fd16cd948d5122305a3c2dbf2f6b7558f1c69372"
Oct 30 19:45:05.970127 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:05.970100    4712 generic.go:356] "Generic (PLEG): container finished" podID="e7126065-03f2-422f-ace7-566b947ed91a" containerID="fd959933f9efb1a21a64612fcf12a0c6b2c5ed879659f2fda6e23986731164e5" exitCode=60
Oct 30 19:45:05.971906 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:05.970329    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl" event={"ID":"e7126065-03f2-422f-ace7-566b947ed91a","Type":"ContainerDied","Data":"fd959933f9efb1a21a64612fcf12a0c6b2c5ed879659f2fda6e23986731164e5"}
Oct 30 19:45:05.972085 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:05.970661    4712 scope.go:122] "RemoveContainer" containerID="fd959933f9efb1a21a64612fcf12a0c6b2c5ed879659f2fda6e23986731164e5"
Oct 30 19:45:05.975046 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:05.975025    4712 generic.go:356] "Generic (PLEG): container finished" podID="4ae23e2d-5493-45ce-b64a-42e8bc7badcf" containerID="b977034aec49dbe88dd31bb5062f83277d04a0f9b0c2cf1fcc5d8164d1201ebe" exitCode=60
Oct 30 19:45:05.975225 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:05.975175    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8" event={"ID":"4ae23e2d-5493-45ce-b64a-42e8bc7badcf","Type":"ContainerDied","Data":"b977034aec49dbe88dd31bb5062f83277d04a0f9b0c2cf1fcc5d8164d1201ebe"}
Oct 30 19:45:05.975309 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:05.975296    4712 scope.go:122] "RemoveContainer" containerID="b977034aec49dbe88dd31bb5062f83277d04a0f9b0c2cf1fcc5d8164d1201ebe"
Oct 30 19:45:05.980813 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:05.980794    4712 generic.go:356] "Generic (PLEG): container finished" podID="5c79a999-80c1-44fa-b15b-2230defbbbd2" containerID="e054bb78ce71cd47614ba29ad6325317e6ed4d0641990cfbabe869b1654bfb93" exitCode=60
Oct 30 19:45:05.980924 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:05.980819    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r" event={"ID":"5c79a999-80c1-44fa-b15b-2230defbbbd2","Type":"ContainerDied","Data":"e054bb78ce71cd47614ba29ad6325317e6ed4d0641990cfbabe869b1654bfb93"}
Oct 30 19:45:05.981091 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:05.981076    4712 scope.go:122] "RemoveContainer" containerID="e054bb78ce71cd47614ba29ad6325317e6ed4d0641990cfbabe869b1654bfb93"
Oct 30 19:45:06.115733 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:06.115670    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:06.337024 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:06.336996    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:06.376608 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:06.376568    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:45:06.988924 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:06.988890    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl" event={"ID":"e7126065-03f2-422f-ace7-566b947ed91a","Type":"ContainerStarted","Data":"8ae49b206df0bcd3d2b748405599f69d4ad3c05ee503e67c115b9d52a3b71daa"}
Oct 30 19:45:06.991341 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:06.991003    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8" event={"ID":"4ae23e2d-5493-45ce-b64a-42e8bc7badcf","Type":"ContainerStarted","Data":"7f5a9524a640166a0a7a0727f9b4235c6d8bc5713a6de0cde824554dc46ce2e1"}
Oct 30 19:45:06.996027 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:06.996000    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r" event={"ID":"5c79a999-80c1-44fa-b15b-2230defbbbd2","Type":"ContainerStarted","Data":"683bcb81e2c83edff06be0ea161ff98a00d1c47842bdc91eda7fed10c28cc442"}
Oct 30 19:45:06.998188 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:06.998164    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch" event={"ID":"0c6a1252-2465-41d9-9a09-96489fd30f66","Type":"ContainerStarted","Data":"e885b32239295db8c08511d4ba17e82b352ecff938e34cace41808ccff0f5a4a"}
Oct 30 19:45:07.036781 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:07.036726    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl" podStartSLOduration=3.713331911 podStartE2EDuration="7.036708571s" totalImagesPullingTime="3.32337666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:07.036071592 +0000 UTC m=+1460.335333723" watchObservedRunningTime="2026-10-30 19:45:07.036708571 +0000 UTC m=+1460.335970700"
Oct 30 19:45:07.054138 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:07.054113    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:07.087985 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:07.086545    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8" podStartSLOduration=3.607290719 podStartE2EDuration="7.086525762s" totalImagesPullingTime="3.479235043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:07.0811307 +0000 UTC m=+1460.380392830" watchObservedRunningTime="2026-10-30 19:45:07.086525762 +0000 UTC m=+1460.385787891"
Oct 30 19:45:07.126724 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:07.126680    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch" podStartSLOduration=3.787404491 podStartE2EDuration="7.126666944s" totalImagesPullingTime="3.339262453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:07.125745551 +0000 UTC m=+1460.425007681" watchObservedRunningTime="2026-10-30 19:45:07.126666944 +0000 UTC m=+1460.425929073"
Oct 30 19:45:07.222464 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:07.222440    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:07.572701 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:07.572614    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:08.006588 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:08.006555    4712 generic.go:356] "Generic (PLEG): container finished" podID="e7126065-03f2-422f-ace7-566b947ed91a" containerID="8ae49b206df0bcd3d2b748405599f69d4ad3c05ee503e67c115b9d52a3b71daa" exitCode=60
Oct 30 19:45:08.007045 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:08.006632    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl" event={"ID":"e7126065-03f2-422f-ace7-566b947ed91a","Type":"ContainerDied","Data":"8ae49b206df0bcd3d2b748405599f69d4ad3c05ee503e67c115b9d52a3b71daa"}
Oct 30 19:45:08.007045 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:08.006689    4712 scope.go:122] "RemoveContainer" containerID="fd959933f9efb1a21a64612fcf12a0c6b2c5ed879659f2fda6e23986731164e5"
Oct 30 19:45:08.007045 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:08.006888    4712 scope.go:122] "RemoveContainer" containerID="8ae49b206df0bcd3d2b748405599f69d4ad3c05ee503e67c115b9d52a3b71daa"
Oct 30 19:45:08.007288 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:08.007264    4712 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl_odh-ai-gateway-infra(e7126065-03f2-422f-ace7-566b947ed91a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl" podUID="e7126065-03f2-422f-ace7-566b947ed91a"
Oct 30 19:45:08.008818 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:08.008796    4712 generic.go:356] "Generic (PLEG): container finished" podID="4ae23e2d-5493-45ce-b64a-42e8bc7badcf" containerID="7f5a9524a640166a0a7a0727f9b4235c6d8bc5713a6de0cde824554dc46ce2e1" exitCode=60
Oct 30 19:45:08.008920 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:08.008874    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8" event={"ID":"4ae23e2d-5493-45ce-b64a-42e8bc7badcf","Type":"ContainerDied","Data":"7f5a9524a640166a0a7a0727f9b4235c6d8bc5713a6de0cde824554dc46ce2e1"}
Oct 30 19:45:08.009084 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:08.009069    4712 scope.go:122] "RemoveContainer" containerID="7f5a9524a640166a0a7a0727f9b4235c6d8bc5713a6de0cde824554dc46ce2e1"
Oct 30 19:45:08.009402 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:08.009378    4712 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8_odh-ai-gateway-infra(4ae23e2d-5493-45ce-b64a-42e8bc7badcf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8" podUID="4ae23e2d-5493-45ce-b64a-42e8bc7badcf"
Oct 30 19:45:08.010750 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:08.010730    4712 generic.go:356] "Generic (PLEG): container finished" podID="5c79a999-80c1-44fa-b15b-2230defbbbd2" containerID="683bcb81e2c83edff06be0ea161ff98a00d1c47842bdc91eda7fed10c28cc442" exitCode=60
Oct 30 19:45:08.010829 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:08.010809    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r" event={"ID":"5c79a999-80c1-44fa-b15b-2230defbbbd2","Type":"ContainerDied","Data":"683bcb81e2c83edff06be0ea161ff98a00d1c47842bdc91eda7fed10c28cc442"}
Oct 30 19:45:08.011183 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:08.011167    4712 scope.go:122] "RemoveContainer" containerID="683bcb81e2c83edff06be0ea161ff98a00d1c47842bdc91eda7fed10c28cc442"
Oct 30 19:45:08.011517 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:08.011499    4712 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r_odh-ai-gateway-infra(5c79a999-80c1-44fa-b15b-2230defbbbd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r" podUID="5c79a999-80c1-44fa-b15b-2230defbbbd2"
Oct 30 19:45:08.012757 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:08.012733    4712 generic.go:356] "Generic (PLEG): container finished" podID="0c6a1252-2465-41d9-9a09-96489fd30f66" containerID="e885b32239295db8c08511d4ba17e82b352ecff938e34cace41808ccff0f5a4a" exitCode=60
Oct 30 19:45:08.012824 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:08.012774    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch" event={"ID":"0c6a1252-2465-41d9-9a09-96489fd30f66","Type":"ContainerDied","Data":"e885b32239295db8c08511d4ba17e82b352ecff938e34cace41808ccff0f5a4a"}
Oct 30 19:45:08.013036 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:08.013023    4712 scope.go:122] "RemoveContainer" containerID="e885b32239295db8c08511d4ba17e82b352ecff938e34cace41808ccff0f5a4a"
Oct 30 19:45:08.013332 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:08.013311    4712 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch_odh-ai-gateway-infra(0c6a1252-2465-41d9-9a09-96489fd30f66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch" podUID="0c6a1252-2465-41d9-9a09-96489fd30f66"
Oct 30 19:45:08.019849 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:08.019833    4712 scope.go:122] "RemoveContainer" containerID="b977034aec49dbe88dd31bb5062f83277d04a0f9b0c2cf1fcc5d8164d1201ebe"
Oct 30 19:45:08.034640 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:08.034599    4712 scope.go:122] "RemoveContainer" containerID="e054bb78ce71cd47614ba29ad6325317e6ed4d0641990cfbabe869b1654bfb93"
Oct 30 19:45:08.047920 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:08.047869    4712 scope.go:122] "RemoveContainer" containerID="576d9492a031c07438bee615fd16cd948d5122305a3c2dbf2f6b7558f1c69372"
Oct 30 19:45:09.021715 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:09.021685    4712 scope.go:122] "RemoveContainer" containerID="7f5a9524a640166a0a7a0727f9b4235c6d8bc5713a6de0cde824554dc46ce2e1"
Oct 30 19:45:09.022176 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:09.022038    4712 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8_odh-ai-gateway-infra(4ae23e2d-5493-45ce-b64a-42e8bc7badcf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8" podUID="4ae23e2d-5493-45ce-b64a-42e8bc7badcf"
Oct 30 19:45:09.025298 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:09.025274    4712 scope.go:122] "RemoveContainer" containerID="e885b32239295db8c08511d4ba17e82b352ecff938e34cace41808ccff0f5a4a"
Oct 30 19:45:09.025626 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:09.025591    4712 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch_odh-ai-gateway-infra(0c6a1252-2465-41d9-9a09-96489fd30f66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch" podUID="0c6a1252-2465-41d9-9a09-96489fd30f66"
Oct 30 19:45:09.028561 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:09.028539    4712 scope.go:122] "RemoveContainer" containerID="8ae49b206df0bcd3d2b748405599f69d4ad3c05ee503e67c115b9d52a3b71daa"
Oct 30 19:45:09.028812 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:09.028786    4712 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl_odh-ai-gateway-infra(e7126065-03f2-422f-ace7-566b947ed91a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl" podUID="e7126065-03f2-422f-ace7-566b947ed91a"
Oct 30 19:45:09.985430 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:09.985385    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:10.201256 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:10.201202    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:10.402757 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:10.402730    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:11.436442 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:11.436413    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:11.659163 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:11.659133    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:11.769597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:11.769520    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"]
Oct 30 19:45:11.769868 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:11.769842    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57" podUID="bcdfb861-99cb-4c80-882f-fed7871177c2" containerName="istio-proxy" containerID="cri-o://88a50754e8ee0324f4dd9c846ace2d19f8569e0a86bc1e7ddf44ab302c8abb8f" gracePeriod=30
Oct 30 19:45:11.781268 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:11.781245    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"]
Oct 30 19:45:11.781476 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:11.781455    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm" podUID="16005c00-9c89-4553-aa62-6d81b4243851" containerName="istio-proxy" containerID="cri-o://fe4005ee64917e85be24c507f0a180b134d3cde1e07cb1c0b4ea767d179721cf" gracePeriod=30
Oct 30 19:45:11.987460 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:11.987230    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:12.259087 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:12.259051    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:13.791614 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:13.791585    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:14.344616 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:14.344586    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:15.295961 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:15.295924    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:17.074564 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.074491    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:45:17.089097 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.089072    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-envoy\") pod \"bcdfb861-99cb-4c80-882f-fed7871177c2\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") "
Oct 30 19:45:17.089208 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.089117    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-data\") pod \"bcdfb861-99cb-4c80-882f-fed7871177c2\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") "
Oct 30 19:45:17.089299 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.089278    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcdfb861-99cb-4c80-882f-fed7871177c2-kube-api-access-rpvk2\" (UniqueName: \"kubernetes.io/projected/bcdfb861-99cb-4c80-882f-fed7871177c2-kube-api-access-rpvk2\") pod \"bcdfb861-99cb-4c80-882f-fed7871177c2\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") "
Oct 30 19:45:17.089784 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.089352    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-token\" (UniqueName: \"kubernetes.io/projected/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-token\") pod \"bcdfb861-99cb-4c80-882f-fed7871177c2\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") "
Oct 30 19:45:17.089784 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.089405    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-podinfo\") pod \"bcdfb861-99cb-4c80-882f-fed7871177c2\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") "
Oct 30 19:45:17.089784 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.089514    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-data" pod "bcdfb861-99cb-4c80-882f-fed7871177c2" (UID: "bcdfb861-99cb-4c80-882f-fed7871177c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:17.090104 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.090075    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.091512 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.091481    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-envoy" pod "bcdfb861-99cb-4c80-882f-fed7871177c2" (UID: "bcdfb861-99cb-4c80-882f-fed7871177c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:17.092515 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.092493    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-token" pod "bcdfb861-99cb-4c80-882f-fed7871177c2" (UID: "bcdfb861-99cb-4c80-882f-fed7871177c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:17.092623 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.092574    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-podinfo" pod "bcdfb861-99cb-4c80-882f-fed7871177c2" (UID: "bcdfb861-99cb-4c80-882f-fed7871177c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:45:17.093055 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.093039    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcdfb861-99cb-4c80-882f-fed7871177c2-kube-api-access-rpvk2" pod "bcdfb861-99cb-4c80-882f-fed7871177c2" (UID: "bcdfb861-99cb-4c80-882f-fed7871177c2"). InnerVolumeSpecName "kube-api-access-rpvk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:17.113174 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.113157    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:45:17.125326 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.125301    4712 generic.go:356] "Generic (PLEG): container finished" podID="bcdfb861-99cb-4c80-882f-fed7871177c2" containerID="88a50754e8ee0324f4dd9c846ace2d19f8569e0a86bc1e7ddf44ab302c8abb8f" exitCode=0
Oct 30 19:45:17.125446 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.125420    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57" event={"ID":"bcdfb861-99cb-4c80-882f-fed7871177c2","Type":"ContainerDied","Data":"88a50754e8ee0324f4dd9c846ace2d19f8569e0a86bc1e7ddf44ab302c8abb8f"}
Oct 30 19:45:17.125524 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.125473    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57" event={"ID":"bcdfb861-99cb-4c80-882f-fed7871177c2","Type":"ContainerDied","Data":"7d3a9b178997a540c8a0f786973b3c8705f20befd960de09f5c89d284e05d31b"}
Oct 30 19:45:17.125524 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.125505    4712 scope.go:122] "RemoveContainer" containerID="88a50754e8ee0324f4dd9c846ace2d19f8569e0a86bc1e7ddf44ab302c8abb8f"
Oct 30 19:45:17.125667 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.125652    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"
Oct 30 19:45:17.127483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.127325    4712 generic.go:356] "Generic (PLEG): container finished" podID="16005c00-9c89-4553-aa62-6d81b4243851" containerID="fe4005ee64917e85be24c507f0a180b134d3cde1e07cb1c0b4ea767d179721cf" exitCode=0
Oct 30 19:45:17.127483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.127389    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"
Oct 30 19:45:17.127483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.127403    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm" event={"ID":"16005c00-9c89-4553-aa62-6d81b4243851","Type":"ContainerDied","Data":"fe4005ee64917e85be24c507f0a180b134d3cde1e07cb1c0b4ea767d179721cf"}
Oct 30 19:45:17.127483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.127464    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm" event={"ID":"16005c00-9c89-4553-aa62-6d81b4243851","Type":"ContainerDied","Data":"104c0e7e54bd2ad215b07aa69440b234a274941f18239a44bce5fe2da4655211"}
Oct 30 19:45:17.136225 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.136206    4712 scope.go:122] "RemoveContainer" containerID="88a50754e8ee0324f4dd9c846ace2d19f8569e0a86bc1e7ddf44ab302c8abb8f"
Oct 30 19:45:17.136593 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:17.136559    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88a50754e8ee0324f4dd9c846ace2d19f8569e0a86bc1e7ddf44ab302c8abb8f\": container with ID starting with 88a50754e8ee0324f4dd9c846ace2d19f8569e0a86bc1e7ddf44ab302c8abb8f not found: ID does not exist" containerID="88a50754e8ee0324f4dd9c846ace2d19f8569e0a86bc1e7ddf44ab302c8abb8f"
Oct 30 19:45:17.136688 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.136602    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a50754e8ee0324f4dd9c846ace2d19f8569e0a86bc1e7ddf44ab302c8abb8f"} err="failed to get container status \"88a50754e8ee0324f4dd9c846ace2d19f8569e0a86bc1e7ddf44ab302c8abb8f\": rpc error: code = NotFound desc = could not find container \"88a50754e8ee0324f4dd9c846ace2d19f8569e0a86bc1e7ddf44ab302c8abb8f\": container with ID starting with 88a50754e8ee0324f4dd9c846ace2d19f8569e0a86bc1e7ddf44ab302c8abb8f not found: ID does not exist"
Oct 30 19:45:17.136688 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.136626    4712 scope.go:122] "RemoveContainer" containerID="fe4005ee64917e85be24c507f0a180b134d3cde1e07cb1c0b4ea767d179721cf"
Oct 30 19:45:17.146542 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.146520    4712 scope.go:122] "RemoveContainer" containerID="fe4005ee64917e85be24c507f0a180b134d3cde1e07cb1c0b4ea767d179721cf"
Oct 30 19:45:17.146851 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:17.146828    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe4005ee64917e85be24c507f0a180b134d3cde1e07cb1c0b4ea767d179721cf\": container with ID starting with fe4005ee64917e85be24c507f0a180b134d3cde1e07cb1c0b4ea767d179721cf not found: ID does not exist" containerID="fe4005ee64917e85be24c507f0a180b134d3cde1e07cb1c0b4ea767d179721cf"
Oct 30 19:45:17.146966 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.146858    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe4005ee64917e85be24c507f0a180b134d3cde1e07cb1c0b4ea767d179721cf"} err="failed to get container status \"fe4005ee64917e85be24c507f0a180b134d3cde1e07cb1c0b4ea767d179721cf\": rpc error: code = NotFound desc = could not find container \"fe4005ee64917e85be24c507f0a180b134d3cde1e07cb1c0b4ea767d179721cf\": container with ID starting with fe4005ee64917e85be24c507f0a180b134d3cde1e07cb1c0b4ea767d179721cf not found: ID does not exist"
Oct 30 19:45:17.191307 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.191277    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16005c00-9c89-4553-aa62-6d81b4243851-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16005c00-9c89-4553-aa62-6d81b4243851-istio-podinfo\") pod \"16005c00-9c89-4553-aa62-6d81b4243851\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") "
Oct 30 19:45:17.191440 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.191403    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-credential-socket\") pod \"16005c00-9c89-4553-aa62-6d81b4243851\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") "
Oct 30 19:45:17.191502 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.191447    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-workload-certs\") pod \"bcdfb861-99cb-4c80-882f-fed7871177c2\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") "
Oct 30 19:45:17.191559 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.191501    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-credential-socket\") pod \"bcdfb861-99cb-4c80-882f-fed7871177c2\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") "
Oct 30 19:45:17.191559 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.191531    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-workload-socket\") pod \"bcdfb861-99cb-4c80-882f-fed7871177c2\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") "
Oct 30 19:45:17.191672 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.191641    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16005c00-9c89-4553-aa62-6d81b4243851-istio-token\" (UniqueName: \"kubernetes.io/projected/16005c00-9c89-4553-aa62-6d81b4243851-istio-token\") pod \"16005c00-9c89-4553-aa62-6d81b4243851\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") "
Oct 30 19:45:17.191733 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.191679    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcdfb861-99cb-4c80-882f-fed7871177c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcdfb861-99cb-4c80-882f-fed7871177c2-istiod-ca-cert\") pod \"bcdfb861-99cb-4c80-882f-fed7871177c2\" (UID: \"bcdfb861-99cb-4c80-882f-fed7871177c2\") "
Oct 30 19:45:17.191839 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.191732    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-workload-socket\") pod \"16005c00-9c89-4553-aa62-6d81b4243851\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") "
Oct 30 19:45:17.191839 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.191736    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-workload-certs" pod "bcdfb861-99cb-4c80-882f-fed7871177c2" (UID: "bcdfb861-99cb-4c80-882f-fed7871177c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:17.191839 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.191748    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-credential-socket" pod "16005c00-9c89-4553-aa62-6d81b4243851" (UID: "16005c00-9c89-4553-aa62-6d81b4243851"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:17.192353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.191854    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-workload-socket" pod "bcdfb861-99cb-4c80-882f-fed7871177c2" (UID: "bcdfb861-99cb-4c80-882f-fed7871177c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:17.192353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.192006    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-istio-data\") pod \"16005c00-9c89-4553-aa62-6d81b4243851\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") "
Oct 30 19:45:17.192353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.192027    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-workload-socket" pod "16005c00-9c89-4553-aa62-6d81b4243851" (UID: "16005c00-9c89-4553-aa62-6d81b4243851"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:17.192353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.192110    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16005c00-9c89-4553-aa62-6d81b4243851-kube-api-access-vn7kl\" (UniqueName: \"kubernetes.io/projected/16005c00-9c89-4553-aa62-6d81b4243851-kube-api-access-vn7kl\") pod \"16005c00-9c89-4553-aa62-6d81b4243851\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") "
Oct 30 19:45:17.192353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.192196    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-workload-certs\") pod \"16005c00-9c89-4553-aa62-6d81b4243851\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") "
Oct 30 19:45:17.192353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.192200    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-credential-socket" pod "bcdfb861-99cb-4c80-882f-fed7871177c2" (UID: "bcdfb861-99cb-4c80-882f-fed7871177c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:17.192353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.192225    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16005c00-9c89-4553-aa62-6d81b4243851-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16005c00-9c89-4553-aa62-6d81b4243851-istiod-ca-cert\") pod \"16005c00-9c89-4553-aa62-6d81b4243851\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") "
Oct 30 19:45:17.192353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.192220    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcdfb861-99cb-4c80-882f-fed7871177c2-istiod-ca-cert" pod "bcdfb861-99cb-4c80-882f-fed7871177c2" (UID: "bcdfb861-99cb-4c80-882f-fed7871177c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:17.192353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.192341    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-istio-data" pod "16005c00-9c89-4553-aa62-6d81b4243851" (UID: "16005c00-9c89-4553-aa62-6d81b4243851"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:17.192814 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.192457    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-workload-certs" pod "16005c00-9c89-4553-aa62-6d81b4243851" (UID: "16005c00-9c89-4553-aa62-6d81b4243851"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:17.192814 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.192464    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-istio-envoy\") pod \"16005c00-9c89-4553-aa62-6d81b4243851\" (UID: \"16005c00-9c89-4553-aa62-6d81b4243851\") "
Oct 30 19:45:17.192982 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.192960    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16005c00-9c89-4553-aa62-6d81b4243851-istiod-ca-cert" pod "16005c00-9c89-4553-aa62-6d81b4243851" (UID: "16005c00-9c89-4553-aa62-6d81b4243851"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:17.193114 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.193099    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.193168 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.193115    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16005c00-9c89-4553-aa62-6d81b4243851-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.193168 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.193125    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpvk2\" (UniqueName: \"kubernetes.io/projected/bcdfb861-99cb-4c80-882f-fed7871177c2-kube-api-access-rpvk2\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.193168 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.193134    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.193168 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.193142    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.193168 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.193150    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.193168 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.193159    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.193168 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.193166    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.193566 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.193176    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.193566 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.193185    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcdfb861-99cb-4c80-882f-fed7871177c2-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.193566 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.193194    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.193566 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.193203    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcdfb861-99cb-4c80-882f-fed7871177c2-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.193566 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.193211    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.194587 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.194565    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16005c00-9c89-4553-aa62-6d81b4243851-istio-token" pod "16005c00-9c89-4553-aa62-6d81b4243851" (UID: "16005c00-9c89-4553-aa62-6d81b4243851"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:17.194678 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.194596    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16005c00-9c89-4553-aa62-6d81b4243851-kube-api-access-vn7kl" pod "16005c00-9c89-4553-aa62-6d81b4243851" (UID: "16005c00-9c89-4553-aa62-6d81b4243851"). InnerVolumeSpecName "kube-api-access-vn7kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:17.194726 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.194679    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16005c00-9c89-4553-aa62-6d81b4243851-istio-podinfo" pod "16005c00-9c89-4553-aa62-6d81b4243851" (UID: "16005c00-9c89-4553-aa62-6d81b4243851"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:45:17.194909 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.194897    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-istio-envoy" pod "16005c00-9c89-4553-aa62-6d81b4243851" (UID: "16005c00-9c89-4553-aa62-6d81b4243851"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:17.294776 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.294739    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16005c00-9c89-4553-aa62-6d81b4243851-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.294776 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.294776    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn7kl\" (UniqueName: \"kubernetes.io/projected/16005c00-9c89-4553-aa62-6d81b4243851-kube-api-access-vn7kl\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.295034 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.294789    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16005c00-9c89-4553-aa62-6d81b4243851-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.295034 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.294801    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16005c00-9c89-4553-aa62-6d81b4243851-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:17.503885 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.503853    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"]
Oct 30 19:45:17.516920 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.516890    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-kdz57"]
Oct 30 19:45:17.535429 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.535398    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:17.569414 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.569376    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"]
Oct 30 19:45:17.576526 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:17.576499    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd23d1-openshift-default-8f47ff88c-7wcgm"]
Oct 30 19:45:18.846117 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:18.846085    4712 scope.go:122] "RemoveContainer" containerID="e885b32239295db8c08511d4ba17e82b352ecff938e34cace41808ccff0f5a4a"
Oct 30 19:45:18.856091 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:18.856060    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16005c00-9c89-4553-aa62-6d81b4243851" path="/var/lib/kubelet/pods/16005c00-9c89-4553-aa62-6d81b4243851/volumes"
Oct 30 19:45:18.856730 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:18.856711    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcdfb861-99cb-4c80-882f-fed7871177c2" path="/var/lib/kubelet/pods/bcdfb861-99cb-4c80-882f-fed7871177c2/volumes"
Oct 30 19:45:19.144012 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:19.143988    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch" event={"ID":"0c6a1252-2465-41d9-9a09-96489fd30f66","Type":"ContainerStarted","Data":"a523d16a8bb07f9712aa9f7d3d95dfcd72fd36eab0a7b0265276557951faaaf0"}
Oct 30 19:45:19.335907 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:19.335874    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:19.845282 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:19.845253    4712 scope.go:122] "RemoveContainer" containerID="8ae49b206df0bcd3d2b748405599f69d4ad3c05ee503e67c115b9d52a3b71daa"
Oct 30 19:45:20.152251 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:20.152211    4712 generic.go:356] "Generic (PLEG): container finished" podID="0c6a1252-2465-41d9-9a09-96489fd30f66" containerID="a523d16a8bb07f9712aa9f7d3d95dfcd72fd36eab0a7b0265276557951faaaf0" exitCode=60
Oct 30 19:45:20.152732 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:20.152280    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch" event={"ID":"0c6a1252-2465-41d9-9a09-96489fd30f66","Type":"ContainerDied","Data":"a523d16a8bb07f9712aa9f7d3d95dfcd72fd36eab0a7b0265276557951faaaf0"}
Oct 30 19:45:20.152732 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:20.152335    4712 scope.go:122] "RemoveContainer" containerID="e885b32239295db8c08511d4ba17e82b352ecff938e34cace41808ccff0f5a4a"
Oct 30 19:45:20.152872 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:20.152736    4712 scope.go:122] "RemoveContainer" containerID="a523d16a8bb07f9712aa9f7d3d95dfcd72fd36eab0a7b0265276557951faaaf0"
Oct 30 19:45:20.153084 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:20.153061    4712 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch_odh-ai-gateway-infra(0c6a1252-2465-41d9-9a09-96489fd30f66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch" podUID="0c6a1252-2465-41d9-9a09-96489fd30f66"
Oct 30 19:45:20.155231 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:20.155210    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl" event={"ID":"e7126065-03f2-422f-ace7-566b947ed91a","Type":"ContainerStarted","Data":"c4b43207353045f0b011c86740bbb414a6965bddbe0ea83848ecb76289a8fb16"}
Oct 30 19:45:20.208216 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:20.208180    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch"]
Oct 30 19:45:20.845531 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:20.845508    4712 scope.go:122] "RemoveContainer" containerID="683bcb81e2c83edff06be0ea161ff98a00d1c47842bdc91eda7fed10c28cc442"
Oct 30 19:45:20.845861 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:20.845838    4712 scope.go:122] "RemoveContainer" containerID="7f5a9524a640166a0a7a0727f9b4235c6d8bc5713a6de0cde824554dc46ce2e1"
Oct 30 19:45:21.171028 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:21.171000    4712 generic.go:356] "Generic (PLEG): container finished" podID="e7126065-03f2-422f-ace7-566b947ed91a" containerID="c4b43207353045f0b011c86740bbb414a6965bddbe0ea83848ecb76289a8fb16" exitCode=60
Oct 30 19:45:21.171432 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:21.171068    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl" event={"ID":"e7126065-03f2-422f-ace7-566b947ed91a","Type":"ContainerDied","Data":"c4b43207353045f0b011c86740bbb414a6965bddbe0ea83848ecb76289a8fb16"}
Oct 30 19:45:21.171432 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:21.171126    4712 scope.go:122] "RemoveContainer" containerID="8ae49b206df0bcd3d2b748405599f69d4ad3c05ee503e67c115b9d52a3b71daa"
Oct 30 19:45:21.171520 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:21.171471    4712 scope.go:122] "RemoveContainer" containerID="c4b43207353045f0b011c86740bbb414a6965bddbe0ea83848ecb76289a8fb16"
Oct 30 19:45:21.171817 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:21.171795    4712 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl_odh-ai-gateway-infra(e7126065-03f2-422f-ace7-566b947ed91a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl" podUID="e7126065-03f2-422f-ace7-566b947ed91a"
Oct 30 19:45:21.272803 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:21.272754    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:21.275506 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:21.275476    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl"]
Oct 30 19:45:21.372863 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:21.372814    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch"
Oct 30 19:45:21.458614 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:21.458576    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c6a1252-2465-41d9-9a09-96489fd30f66-kube-api-access-mzbw9\" (UniqueName: \"kubernetes.io/projected/0c6a1252-2465-41d9-9a09-96489fd30f66-kube-api-access-mzbw9\") pod \"0c6a1252-2465-41d9-9a09-96489fd30f66\" (UID: \"0c6a1252-2465-41d9-9a09-96489fd30f66\") "
Oct 30 19:45:21.458855 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:21.458834    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c6a1252-2465-41d9-9a09-96489fd30f66-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6a1252-2465-41d9-9a09-96489fd30f66-maas-api-service-ca\") pod \"0c6a1252-2465-41d9-9a09-96489fd30f66\" (UID: \"0c6a1252-2465-41d9-9a09-96489fd30f66\") "
Oct 30 19:45:21.459174 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:21.459150    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6a1252-2465-41d9-9a09-96489fd30f66-maas-api-service-ca" pod "0c6a1252-2465-41d9-9a09-96489fd30f66" (UID: "0c6a1252-2465-41d9-9a09-96489fd30f66"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:21.459523 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:21.459506    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6a1252-2465-41d9-9a09-96489fd30f66-maas-api-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:21.460465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:21.460444    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6a1252-2465-41d9-9a09-96489fd30f66-kube-api-access-mzbw9" pod "0c6a1252-2465-41d9-9a09-96489fd30f66" (UID: "0c6a1252-2465-41d9-9a09-96489fd30f66"). InnerVolumeSpecName "kube-api-access-mzbw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:21.561309 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:21.561276    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzbw9\" (UniqueName: \"kubernetes.io/projected/0c6a1252-2465-41d9-9a09-96489fd30f66-kube-api-access-mzbw9\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:22.178567 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.178534    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch"
Oct 30 19:45:22.178567 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.178545    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch" event={"ID":"0c6a1252-2465-41d9-9a09-96489fd30f66","Type":"ContainerDied","Data":"6b9c92efb4464a891438ee8aa8312d42ea60f5690a08b7f6ad9f000fe47a0633"}
Oct 30 19:45:22.179054 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.178609    4712 scope.go:122] "RemoveContainer" containerID="a523d16a8bb07f9712aa9f7d3d95dfcd72fd36eab0a7b0265276557951faaaf0"
Oct 30 19:45:22.182913 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.182878    4712 generic.go:356] "Generic (PLEG): container finished" podID="4ae23e2d-5493-45ce-b64a-42e8bc7badcf" containerID="1304e3ac350afeb9c683f6a90c3b1107f53fc8ea9acf8e9e6a672d77e9858798" exitCode=60
Oct 30 19:45:22.183026 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.182943    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8" event={"ID":"4ae23e2d-5493-45ce-b64a-42e8bc7badcf","Type":"ContainerDied","Data":"1304e3ac350afeb9c683f6a90c3b1107f53fc8ea9acf8e9e6a672d77e9858798"}
Oct 30 19:45:22.183405 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.183385    4712 scope.go:122] "RemoveContainer" containerID="1304e3ac350afeb9c683f6a90c3b1107f53fc8ea9acf8e9e6a672d77e9858798"
Oct 30 19:45:22.183727 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:22.183704    4712 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8_odh-ai-gateway-infra(4ae23e2d-5493-45ce-b64a-42e8bc7badcf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8" podUID="4ae23e2d-5493-45ce-b64a-42e8bc7badcf"
Oct 30 19:45:22.185172 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.185153    4712 generic.go:356] "Generic (PLEG): container finished" podID="5c79a999-80c1-44fa-b15b-2230defbbbd2" containerID="478b1382d84b15c06cd8d65cb301193cdb7fac40dd05b980483d8164f4261bb4" exitCode=60
Oct 30 19:45:22.185241 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.185206    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r" event={"ID":"5c79a999-80c1-44fa-b15b-2230defbbbd2","Type":"ContainerDied","Data":"478b1382d84b15c06cd8d65cb301193cdb7fac40dd05b980483d8164f4261bb4"}
Oct 30 19:45:22.185553 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.185539    4712 scope.go:122] "RemoveContainer" containerID="478b1382d84b15c06cd8d65cb301193cdb7fac40dd05b980483d8164f4261bb4"
Oct 30 19:45:22.185795 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:22.185771    4712 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r_odh-ai-gateway-infra(5c79a999-80c1-44fa-b15b-2230defbbbd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r" podUID="5c79a999-80c1-44fa-b15b-2230defbbbd2"
Oct 30 19:45:22.190984 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.190969    4712 scope.go:122] "RemoveContainer" containerID="7f5a9524a640166a0a7a0727f9b4235c6d8bc5713a6de0cde824554dc46ce2e1"
Oct 30 19:45:22.204439 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.204418    4712 scope.go:122] "RemoveContainer" containerID="683bcb81e2c83edff06be0ea161ff98a00d1c47842bdc91eda7fed10c28cc442"
Oct 30 19:45:22.310448 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.309568    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:22.312461 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.312435    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch"]
Oct 30 19:45:22.317736 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.317715    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-e0e300-29889825-d8hch"]
Oct 30 19:45:22.369886 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.369867    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl"
Oct 30 19:45:22.504938 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.504861    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7126065-03f2-422f-ace7-566b947ed91a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7126065-03f2-422f-ace7-566b947ed91a-maas-api-service-ca\") pod \"e7126065-03f2-422f-ace7-566b947ed91a\" (UID: \"e7126065-03f2-422f-ace7-566b947ed91a\") "
Oct 30 19:45:22.505097 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.504942    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7126065-03f2-422f-ace7-566b947ed91a-kube-api-access-t55fm\" (UniqueName: \"kubernetes.io/projected/e7126065-03f2-422f-ace7-566b947ed91a-kube-api-access-t55fm\") pod \"e7126065-03f2-422f-ace7-566b947ed91a\" (UID: \"e7126065-03f2-422f-ace7-566b947ed91a\") "
Oct 30 19:45:22.505252 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.505227    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7126065-03f2-422f-ace7-566b947ed91a-maas-api-service-ca" pod "e7126065-03f2-422f-ace7-566b947ed91a" (UID: "e7126065-03f2-422f-ace7-566b947ed91a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:22.505413 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.505398    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7126065-03f2-422f-ace7-566b947ed91a-maas-api-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:22.506928 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.506909    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7126065-03f2-422f-ace7-566b947ed91a-kube-api-access-t55fm" pod "e7126065-03f2-422f-ace7-566b947ed91a" (UID: "e7126065-03f2-422f-ace7-566b947ed91a"). InnerVolumeSpecName "kube-api-access-t55fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:22.606749 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.606724    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t55fm\" (UniqueName: \"kubernetes.io/projected/e7126065-03f2-422f-ace7-566b947ed91a-kube-api-access-t55fm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:22.853456 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:22.853386    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c6a1252-2465-41d9-9a09-96489fd30f66" path="/var/lib/kubelet/pods/0c6a1252-2465-41d9-9a09-96489fd30f66/volumes"
Oct 30 19:45:23.077794 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.077715    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:23.195990 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.195960    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl"
Oct 30 19:45:23.196426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.195988    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl" event={"ID":"e7126065-03f2-422f-ace7-566b947ed91a","Type":"ContainerDied","Data":"b2a5a98e343b23cfcf3b7a42e74242aae3f034509158077bd19358f6e8413114"}
Oct 30 19:45:23.196426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.196050    4712 scope.go:122] "RemoveContainer" containerID="c4b43207353045f0b011c86740bbb414a6965bddbe0ea83848ecb76289a8fb16"
Oct 30 19:45:23.252012 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.251975    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl"]
Oct 30 19:45:23.263754 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.263726    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-544fc8-29889825-kxgxl"]
Oct 30 19:45:23.266978 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.266952    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8"]
Oct 30 19:45:23.388239 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.388208    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r"]
Oct 30 19:45:23.427067 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.427030    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8"
Oct 30 19:45:23.524144 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.524114    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae23e2d-5493-45ce-b64a-42e8bc7badcf-kube-api-access-hlhc9\" (UniqueName: \"kubernetes.io/projected/4ae23e2d-5493-45ce-b64a-42e8bc7badcf-kube-api-access-hlhc9\") pod \"4ae23e2d-5493-45ce-b64a-42e8bc7badcf\" (UID: \"4ae23e2d-5493-45ce-b64a-42e8bc7badcf\") "
Oct 30 19:45:23.524325 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.524249    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ae23e2d-5493-45ce-b64a-42e8bc7badcf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae23e2d-5493-45ce-b64a-42e8bc7badcf-maas-api-service-ca\") pod \"4ae23e2d-5493-45ce-b64a-42e8bc7badcf\" (UID: \"4ae23e2d-5493-45ce-b64a-42e8bc7badcf\") "
Oct 30 19:45:23.524694 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.524663    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae23e2d-5493-45ce-b64a-42e8bc7badcf-maas-api-service-ca" pod "4ae23e2d-5493-45ce-b64a-42e8bc7badcf" (UID: "4ae23e2d-5493-45ce-b64a-42e8bc7badcf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:23.524958 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.524933    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae23e2d-5493-45ce-b64a-42e8bc7badcf-maas-api-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:23.526742 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.526670    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae23e2d-5493-45ce-b64a-42e8bc7badcf-kube-api-access-hlhc9" pod "4ae23e2d-5493-45ce-b64a-42e8bc7badcf" (UID: "4ae23e2d-5493-45ce-b64a-42e8bc7badcf"). InnerVolumeSpecName "kube-api-access-hlhc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:23.576095 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.576075    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r"
Oct 30 19:45:23.627453 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.627337    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlhc9\" (UniqueName: \"kubernetes.io/projected/4ae23e2d-5493-45ce-b64a-42e8bc7badcf-kube-api-access-hlhc9\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:23.728910 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.728839    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c79a999-80c1-44fa-b15b-2230defbbbd2-kube-api-access-wtgdc\" (UniqueName: \"kubernetes.io/projected/5c79a999-80c1-44fa-b15b-2230defbbbd2-kube-api-access-wtgdc\") pod \"5c79a999-80c1-44fa-b15b-2230defbbbd2\" (UID: \"5c79a999-80c1-44fa-b15b-2230defbbbd2\") "
Oct 30 19:45:23.729038 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.728948    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c79a999-80c1-44fa-b15b-2230defbbbd2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c79a999-80c1-44fa-b15b-2230defbbbd2-maas-api-service-ca\") pod \"5c79a999-80c1-44fa-b15b-2230defbbbd2\" (UID: \"5c79a999-80c1-44fa-b15b-2230defbbbd2\") "
Oct 30 19:45:23.729312 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.729282    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c79a999-80c1-44fa-b15b-2230defbbbd2-maas-api-service-ca" pod "5c79a999-80c1-44fa-b15b-2230defbbbd2" (UID: "5c79a999-80c1-44fa-b15b-2230defbbbd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:23.729563 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.729539    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c79a999-80c1-44fa-b15b-2230defbbbd2-maas-api-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:23.731384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.731348    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c79a999-80c1-44fa-b15b-2230defbbbd2-kube-api-access-wtgdc" pod "5c79a999-80c1-44fa-b15b-2230defbbbd2" (UID: "5c79a999-80c1-44fa-b15b-2230defbbbd2"). InnerVolumeSpecName "kube-api-access-wtgdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:23.830948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:23.830917    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtgdc\" (UniqueName: \"kubernetes.io/projected/5c79a999-80c1-44fa-b15b-2230defbbbd2-kube-api-access-wtgdc\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:24.205951 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:24.205921    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8"
Oct 30 19:45:24.206424 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:24.205959    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8" event={"ID":"4ae23e2d-5493-45ce-b64a-42e8bc7badcf","Type":"ContainerDied","Data":"29b6e48334a34be5dda5f98cf049aa3c8cd9278142a4b83387bf5967e62354a6"}
Oct 30 19:45:24.206424 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:24.206018    4712 scope.go:122] "RemoveContainer" containerID="1304e3ac350afeb9c683f6a90c3b1107f53fc8ea9acf8e9e6a672d77e9858798"
Oct 30 19:45:24.207317 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:24.207301    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r"
Oct 30 19:45:24.207317 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:24.207306    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r" event={"ID":"5c79a999-80c1-44fa-b15b-2230defbbbd2","Type":"ContainerDied","Data":"5f3e52a06de4353344cf87f76d24b0fc71c2fabd7661b599c27024d496df55b2"}
Oct 30 19:45:24.215856 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:24.215833    4712 scope.go:122] "RemoveContainer" containerID="478b1382d84b15c06cd8d65cb301193cdb7fac40dd05b980483d8164f4261bb4"
Oct 30 19:45:24.257499 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:24.257475    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r"]
Oct 30 19:45:24.269651 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:24.269632    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0c305f-29889825-fcm8r"]
Oct 30 19:45:24.300113 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:24.300092    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8"]
Oct 30 19:45:24.303083 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:24.303065    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-3c89af-29889825-rzgj8"]
Oct 30 19:45:24.853663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:24.853628    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ae23e2d-5493-45ce-b64a-42e8bc7badcf" path="/var/lib/kubelet/pods/4ae23e2d-5493-45ce-b64a-42e8bc7badcf/volumes"
Oct 30 19:45:24.854134 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:24.854118    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c79a999-80c1-44fa-b15b-2230defbbbd2" path="/var/lib/kubelet/pods/5c79a999-80c1-44fa-b15b-2230defbbbd2/volumes"
Oct 30 19:45:24.854532 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:24.854518    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7126065-03f2-422f-ace7-566b947ed91a" path="/var/lib/kubelet/pods/e7126065-03f2-422f-ace7-566b947ed91a/volumes"
Oct 30 19:45:24.879920 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:24.879882    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:26.506633 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:26.506602    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:27.650665 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:27.650631    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:28.180735 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.180703    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl"]
Oct 30 19:45:28.181294 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181279    4712 cpu_manager.go:414] "containerMap: removing container" podUID="4ae23e2d-5493-45ce-b64a-42e8bc7badcf" containerName="cleanup"
Oct 30 19:45:28.181343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181295    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae23e2d-5493-45ce-b64a-42e8bc7badcf" containerName="cleanup"
Oct 30 19:45:28.181343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181307    4712 cpu_manager.go:414] "containerMap: removing container" podUID="16005c00-9c89-4553-aa62-6d81b4243851" containerName="istio-proxy"
Oct 30 19:45:28.181343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181312    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16005c00-9c89-4553-aa62-6d81b4243851" containerName="istio-proxy"
Oct 30 19:45:28.181343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181324    4712 cpu_manager.go:414] "containerMap: removing container" podUID="e7126065-03f2-422f-ace7-566b947ed91a" containerName="cleanup"
Oct 30 19:45:28.181343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181331    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7126065-03f2-422f-ace7-566b947ed91a" containerName="cleanup"
Oct 30 19:45:28.181343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181340    4712 cpu_manager.go:414] "containerMap: removing container" podUID="0c6a1252-2465-41d9-9a09-96489fd30f66" containerName="cleanup"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181345    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6a1252-2465-41d9-9a09-96489fd30f66" containerName="cleanup"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181373    4712 cpu_manager.go:414] "containerMap: removing container" podUID="e7126065-03f2-422f-ace7-566b947ed91a" containerName="cleanup"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181378    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7126065-03f2-422f-ace7-566b947ed91a" containerName="cleanup"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181388    4712 cpu_manager.go:414] "containerMap: removing container" podUID="bcdfb861-99cb-4c80-882f-fed7871177c2" containerName="istio-proxy"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181392    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcdfb861-99cb-4c80-882f-fed7871177c2" containerName="istio-proxy"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181401    4712 cpu_manager.go:414] "containerMap: removing container" podUID="0c6a1252-2465-41d9-9a09-96489fd30f66" containerName="cleanup"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181406    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6a1252-2465-41d9-9a09-96489fd30f66" containerName="cleanup"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181413    4712 cpu_manager.go:414] "containerMap: removing container" podUID="5c79a999-80c1-44fa-b15b-2230defbbbd2" containerName="cleanup"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181418    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c79a999-80c1-44fa-b15b-2230defbbbd2" containerName="cleanup"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181426    4712 cpu_manager.go:414] "containerMap: removing container" podUID="0c6a1252-2465-41d9-9a09-96489fd30f66" containerName="cleanup"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181430    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6a1252-2465-41d9-9a09-96489fd30f66" containerName="cleanup"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181438    4712 cpu_manager.go:414] "containerMap: removing container" podUID="4ae23e2d-5493-45ce-b64a-42e8bc7badcf" containerName="cleanup"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181442    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae23e2d-5493-45ce-b64a-42e8bc7badcf" containerName="cleanup"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181447    4712 cpu_manager.go:414] "containerMap: removing container" podUID="5c79a999-80c1-44fa-b15b-2230defbbbd2" containerName="cleanup"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181451    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c79a999-80c1-44fa-b15b-2230defbbbd2" containerName="cleanup"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181460    4712 cpu_manager.go:414] "containerMap: removing container" podUID="4ae23e2d-5493-45ce-b64a-42e8bc7badcf" containerName="cleanup"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181464    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae23e2d-5493-45ce-b64a-42e8bc7badcf" containerName="cleanup"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181473    4712 cpu_manager.go:414] "containerMap: removing container" podUID="5c79a999-80c1-44fa-b15b-2230defbbbd2" containerName="cleanup"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181477    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c79a999-80c1-44fa-b15b-2230defbbbd2" containerName="cleanup"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181494    4712 cpu_manager.go:414] "containerMap: removing container" podUID="abb8e01c-a453-450b-a7bb-260879385140" containerName="payload-processing"
Oct 30 19:45:28.181552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181498    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb8e01c-a453-450b-a7bb-260879385140" containerName="payload-processing"
Oct 30 19:45:28.182141 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181596    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb8e01c-a453-450b-a7bb-260879385140" containerName="payload-processing"
Oct 30 19:45:28.182141 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181605    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7126065-03f2-422f-ace7-566b947ed91a" containerName="cleanup"
Oct 30 19:45:28.182141 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181613    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae23e2d-5493-45ce-b64a-42e8bc7badcf" containerName="cleanup"
Oct 30 19:45:28.182141 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181618    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="16005c00-9c89-4553-aa62-6d81b4243851" containerName="istio-proxy"
Oct 30 19:45:28.182141 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181625    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7126065-03f2-422f-ace7-566b947ed91a" containerName="cleanup"
Oct 30 19:45:28.182141 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181630    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6a1252-2465-41d9-9a09-96489fd30f66" containerName="cleanup"
Oct 30 19:45:28.182141 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181635    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7126065-03f2-422f-ace7-566b947ed91a" containerName="cleanup"
Oct 30 19:45:28.182141 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181641    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcdfb861-99cb-4c80-882f-fed7871177c2" containerName="istio-proxy"
Oct 30 19:45:28.182141 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181647    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c79a999-80c1-44fa-b15b-2230defbbbd2" containerName="cleanup"
Oct 30 19:45:28.182141 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181653    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6a1252-2465-41d9-9a09-96489fd30f66" containerName="cleanup"
Oct 30 19:45:28.182141 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181657    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae23e2d-5493-45ce-b64a-42e8bc7badcf" containerName="cleanup"
Oct 30 19:45:28.182141 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181669    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c79a999-80c1-44fa-b15b-2230defbbbd2" containerName="cleanup"
Oct 30 19:45:28.182141 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181676    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae23e2d-5493-45ce-b64a-42e8bc7badcf" containerName="cleanup"
Oct 30 19:45:28.182141 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.181682    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c79a999-80c1-44fa-b15b-2230defbbbd2" containerName="cleanup"
Oct 30 19:45:28.184055 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.184027    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:28.186640 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.186621    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl"
Oct 30 19:45:28.204557 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.204534    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl"]
Oct 30 19:45:28.214548 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.214519    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:28.214693 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.214654    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4qb5\""
Oct 30 19:45:28.285751 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.285711    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b431ef98-7f6b-4e01-9cc1-32525af19139-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl\" (UID: \"b431ef98-7f6b-4e01-9cc1-32525af19139\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl"
Oct 30 19:45:28.285751 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.285753    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94dxz\" (UniqueName: \"kubernetes.io/projected/b431ef98-7f6b-4e01-9cc1-32525af19139-kube-api-access-94dxz\") pod \"maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl\" (UID: \"b431ef98-7f6b-4e01-9cc1-32525af19139\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl"
Oct 30 19:45:28.286069 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.286047    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b431ef98-7f6b-4e01-9cc1-32525af19139-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl\" (UID: \"b431ef98-7f6b-4e01-9cc1-32525af19139\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl"
Oct 30 19:45:28.368133 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.368100    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:28.387800 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.387771    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b431ef98-7f6b-4e01-9cc1-32525af19139-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl\" (UID: \"b431ef98-7f6b-4e01-9cc1-32525af19139\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl"
Oct 30 19:45:28.387950 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.387809    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94dxz\" (UniqueName: \"kubernetes.io/projected/b431ef98-7f6b-4e01-9cc1-32525af19139-kube-api-access-94dxz\") pod \"maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl\" (UID: \"b431ef98-7f6b-4e01-9cc1-32525af19139\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl"
Oct 30 19:45:28.388014 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.387973    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b431ef98-7f6b-4e01-9cc1-32525af19139-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl\" (UID: \"b431ef98-7f6b-4e01-9cc1-32525af19139\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl"
Oct 30 19:45:28.388513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.388488    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b431ef98-7f6b-4e01-9cc1-32525af19139-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl\" (UID: \"b431ef98-7f6b-4e01-9cc1-32525af19139\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl"
Oct 30 19:45:28.390434 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.390410    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b431ef98-7f6b-4e01-9cc1-32525af19139-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl\" (UID: \"b431ef98-7f6b-4e01-9cc1-32525af19139\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl"
Oct 30 19:45:28.411628 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.411606    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94dxz\" (UniqueName: \"kubernetes.io/projected/b431ef98-7f6b-4e01-9cc1-32525af19139-kube-api-access-94dxz\") pod \"maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl\" (UID: \"b431ef98-7f6b-4e01-9cc1-32525af19139\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl"
Oct 30 19:45:28.501827 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:28.501775    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl"
Oct 30 19:45:29.015191 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:45:29.015162    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb431ef98_7f6b_4e01_9cc1_32525af19139.slice/crio-29fe9e8b84d2d12e72a02b45d46e2b3a53cc6de34a00e6d82d4c28dcbe43a328 WatchSource:0}: Error finding container 29fe9e8b84d2d12e72a02b45d46e2b3a53cc6de34a00e6d82d4c28dcbe43a328: Status 404 returned error can't find the container with id 29fe9e8b84d2d12e72a02b45d46e2b3a53cc6de34a00e6d82d4c28dcbe43a328
Oct 30 19:45:29.020332 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:29.020297    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl"]
Oct 30 19:45:29.258672 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:29.258636    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl" event={"ID":"b431ef98-7f6b-4e01-9cc1-32525af19139","Type":"ContainerStarted","Data":"29fe9e8b84d2d12e72a02b45d46e2b3a53cc6de34a00e6d82d4c28dcbe43a328"}
Oct 30 19:45:30.266278 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:30.266188    4712 generic.go:356] "Generic (PLEG): container finished" podID="b431ef98-7f6b-4e01-9cc1-32525af19139" containerID="2f21c5b8bc6c33bd33414588545bd366f0b01530e5f1568ffef19c580d83f923" exitCode=0
Oct 30 19:45:30.266732 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:30.266270    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl" event={"ID":"b431ef98-7f6b-4e01-9cc1-32525af19139","Type":"ContainerDied","Data":"2f21c5b8bc6c33bd33414588545bd366f0b01530e5f1568ffef19c580d83f923"}
Oct 30 19:45:30.716387 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:30.714835    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:31.693059 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:31.693030    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl"
Oct 30 19:45:31.896835 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:31.896746    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b431ef98-7f6b-4e01-9cc1-32525af19139-kube-api-access-94dxz\" (UniqueName: \"kubernetes.io/projected/b431ef98-7f6b-4e01-9cc1-32525af19139-kube-api-access-94dxz\") pod \"b431ef98-7f6b-4e01-9cc1-32525af19139\" (UID: \"b431ef98-7f6b-4e01-9cc1-32525af19139\") "
Oct 30 19:45:31.897017 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:31.896969    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b431ef98-7f6b-4e01-9cc1-32525af19139-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b431ef98-7f6b-4e01-9cc1-32525af19139-maas-api-openshift-ca\") pod \"b431ef98-7f6b-4e01-9cc1-32525af19139\" (UID: \"b431ef98-7f6b-4e01-9cc1-32525af19139\") "
Oct 30 19:45:31.897095 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:31.897077    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b431ef98-7f6b-4e01-9cc1-32525af19139-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b431ef98-7f6b-4e01-9cc1-32525af19139-maas-api-xks-ca\") pod \"b431ef98-7f6b-4e01-9cc1-32525af19139\" (UID: \"b431ef98-7f6b-4e01-9cc1-32525af19139\") "
Oct 30 19:45:31.898049 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:31.898015    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b431ef98-7f6b-4e01-9cc1-32525af19139-maas-api-openshift-ca" pod "b431ef98-7f6b-4e01-9cc1-32525af19139" (UID: "b431ef98-7f6b-4e01-9cc1-32525af19139"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:31.900189 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:31.900158    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b431ef98-7f6b-4e01-9cc1-32525af19139-maas-api-xks-ca" pod "b431ef98-7f6b-4e01-9cc1-32525af19139" (UID: "b431ef98-7f6b-4e01-9cc1-32525af19139"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:31.900498 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:31.900446    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b431ef98-7f6b-4e01-9cc1-32525af19139-kube-api-access-94dxz" pod "b431ef98-7f6b-4e01-9cc1-32525af19139" (UID: "b431ef98-7f6b-4e01-9cc1-32525af19139"). InnerVolumeSpecName "kube-api-access-94dxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:31.998978 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:31.998935    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b431ef98-7f6b-4e01-9cc1-32525af19139-maas-api-openshift-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:31.998978 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:31.998971    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b431ef98-7f6b-4e01-9cc1-32525af19139-maas-api-xks-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:31.999210 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:31.998985    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94dxz\" (UniqueName: \"kubernetes.io/projected/b431ef98-7f6b-4e01-9cc1-32525af19139-kube-api-access-94dxz\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:32.292211 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:32.292119    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl" event={"ID":"b431ef98-7f6b-4e01-9cc1-32525af19139","Type":"ContainerDied","Data":"29fe9e8b84d2d12e72a02b45d46e2b3a53cc6de34a00e6d82d4c28dcbe43a328"}
Oct 30 19:45:32.292211 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:32.292151    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl"
Oct 30 19:45:32.292211 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:32.292203    4712 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29fe9e8b84d2d12e72a02b45d46e2b3a53cc6de34a00e6d82d4c28dcbe43a328"
Oct 30 19:45:32.551593 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:32.551511    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"]
Oct 30 19:45:32.580339 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:32.580312    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"]
Oct 30 19:45:32.770803 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:32.770768    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:33.157565 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:33.157529    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:34.022861 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:34.022828    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:34.254241 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:34.254210    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:35.575892 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:35.575860    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:36.294942 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:36.294908    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:36.374052 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:36.374018    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:45:36.554184 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:36.554103    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:36.738953 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:36.738920    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:37.130005 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:37.129976    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:38.441408 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:38.441353    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:38.849960 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:38.849875    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:39.013707 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:39.013662    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:40.426769 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:40.426734    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:42.345209 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:42.345180    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:43.792283 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:43.792244    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:44.585016 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:44.584971    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:44.777023 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:44.776990    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:44.987003 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:44.986971    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:45.425393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:45.425350    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:46.389189 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:46.389153    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:46.644848 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:46.644768    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:46.712552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:46.712522    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"]
Oct 30 19:45:46.712825 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:46.712800    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh" podUID="b4ea703f-3d59-4204-9630-e5915516c074" containerName="istio-proxy" containerID="cri-o://4719c1e72b41ea6049d29de1aef234688f687bd62d0a11c1f6fead039f9d329c" gracePeriod=30
Oct 30 19:45:46.904703 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:46.896795    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:47.054926 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.054897    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:45:47.057397 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.057355    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:45:47.342191 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.342107    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm"]
Oct 30 19:45:47.342807 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.342784    4712 cpu_manager.go:414] "containerMap: removing container" podUID="e7126065-03f2-422f-ace7-566b947ed91a" containerName="cleanup"
Oct 30 19:45:47.342807 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.342806    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7126065-03f2-422f-ace7-566b947ed91a" containerName="cleanup"
Oct 30 19:45:47.342976 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.342868    4712 cpu_manager.go:414] "containerMap: removing container" podUID="b431ef98-7f6b-4e01-9cc1-32525af19139" containerName="revoke-keys"
Oct 30 19:45:47.342976 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.342876    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b431ef98-7f6b-4e01-9cc1-32525af19139" containerName="revoke-keys"
Oct 30 19:45:47.343049 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.343015    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="b431ef98-7f6b-4e01-9cc1-32525af19139" containerName="revoke-keys"
Oct 30 19:45:47.343049 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.343036    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6a1252-2465-41d9-9a09-96489fd30f66" containerName="cleanup"
Oct 30 19:45:47.346585 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.346567    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm"
Oct 30 19:45:47.372393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.372339    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm"]
Oct 30 19:45:47.392835 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.392715    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:47.410188 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.410159    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:45:47.410388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.410248    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4qb5\""
Oct 30 19:45:47.441234 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.441199    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr8q8\" (UniqueName: \"kubernetes.io/projected/804735e2-421e-40c5-8254-b83c9171ce7c-kube-api-access-rr8q8\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm\" (UID: \"804735e2-421e-40c5-8254-b83c9171ce7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm"
Oct 30 19:45:47.441461 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.441258    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804735e2-421e-40c5-8254-b83c9171ce7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm\" (UID: \"804735e2-421e-40c5-8254-b83c9171ce7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm"
Oct 30 19:45:47.441563 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.441550    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804735e2-421e-40c5-8254-b83c9171ce7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm\" (UID: \"804735e2-421e-40c5-8254-b83c9171ce7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm"
Oct 30 19:45:47.542669 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.542636    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8q8\" (UniqueName: \"kubernetes.io/projected/804735e2-421e-40c5-8254-b83c9171ce7c-kube-api-access-rr8q8\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm\" (UID: \"804735e2-421e-40c5-8254-b83c9171ce7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm"
Oct 30 19:45:47.542845 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.542693    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804735e2-421e-40c5-8254-b83c9171ce7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm\" (UID: \"804735e2-421e-40c5-8254-b83c9171ce7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm"
Oct 30 19:45:47.542845 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.542805    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804735e2-421e-40c5-8254-b83c9171ce7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm\" (UID: \"804735e2-421e-40c5-8254-b83c9171ce7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm"
Oct 30 19:45:47.543480 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.543458    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804735e2-421e-40c5-8254-b83c9171ce7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm\" (UID: \"804735e2-421e-40c5-8254-b83c9171ce7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm"
Oct 30 19:45:47.545565 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.545540    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804735e2-421e-40c5-8254-b83c9171ce7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm\" (UID: \"804735e2-421e-40c5-8254-b83c9171ce7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm"
Oct 30 19:45:47.563596 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.563575    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr8q8\" (UniqueName: \"kubernetes.io/projected/804735e2-421e-40c5-8254-b83c9171ce7c-kube-api-access-rr8q8\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm\" (UID: \"804735e2-421e-40c5-8254-b83c9171ce7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm"
Oct 30 19:45:47.572796 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.572769    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main" containerID="cri-o://c7884fb36ae79ee5584ddbe62cedf8720bc522853580c91b186f2dfdd76498ae" gracePeriod=45
Oct 30 19:45:47.600260 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.600215    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main" containerID="cri-o://6a99b12677b8af0c56071f1f82b636ef5cdf3d052f0674136ce8ac3cae7dca0f" gracePeriod=45
Oct 30 19:45:47.668782 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.668758    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm"
Oct 30 19:45:47.914224 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.914193    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:45:47.945356 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:47.945271    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:45:48.017349 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.017313    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm"]
Oct 30 19:45:48.017600 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:45:48.017569    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804735e2_421e_40c5_8254_b83c9171ce7c.slice/crio-fe6830394e6548b4d7c0b3be19b466cd264e0a9d43c96859be66045c65c5c3e1 WatchSource:0}: Error finding container fe6830394e6548b4d7c0b3be19b466cd264e0a9d43c96859be66045c65c5c3e1: Status 404 returned error can't find the container with id fe6830394e6548b4d7c0b3be19b466cd264e0a9d43c96859be66045c65c5c3e1
Oct 30 19:45:48.054093 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.053987    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-home\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-home\") pod \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") "
Oct 30 19:45:48.054093 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.054055    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-model-cache\") pod \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") "
Oct 30 19:45:48.054252 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.054154    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-home\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-home\") pod \"12d39453-861c-437a-8bbe-e9942b8778d3\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") "
Oct 30 19:45:48.054252 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.054185    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12d39453-861c-437a-8bbe-e9942b8778d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/12d39453-861c-437a-8bbe-e9942b8778d3-tls-certs\") pod \"12d39453-861c-437a-8bbe-e9942b8778d3\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") "
Oct 30 19:45:48.054252 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.054224    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-dshm\") pod \"12d39453-861c-437a-8bbe-e9942b8778d3\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") "
Oct 30 19:45:48.054465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.054265    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-model-cache\") pod \"12d39453-861c-437a-8bbe-e9942b8778d3\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") "
Oct 30 19:45:48.054465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.054302    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-home" pod "4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" (UID: "4d5b13af-b0c4-4343-80ca-5c5e0d0c769f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:48.054465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.054458    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-tmp-dir\") pod \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") "
Oct 30 19:45:48.054631 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.054508    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-kube-api-access-zs7m6\" (UniqueName: \"kubernetes.io/projected/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-kube-api-access-zs7m6\") pod \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") "
Oct 30 19:45:48.054631 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.054528    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-home" pod "12d39453-861c-437a-8bbe-e9942b8778d3" (UID: "12d39453-861c-437a-8bbe-e9942b8778d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:48.054631 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.054580    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d39453-861c-437a-8bbe-e9942b8778d3-kube-api-access-srw2b\" (UniqueName: \"kubernetes.io/projected/12d39453-861c-437a-8bbe-e9942b8778d3-kube-api-access-srw2b\") pod \"12d39453-861c-437a-8bbe-e9942b8778d3\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") "
Oct 30 19:45:48.054631 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.054617    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-dshm\") pod \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") "
Oct 30 19:45:48.054829 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.054704    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-model-cache" pod "4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" (UID: "4d5b13af-b0c4-4343-80ca-5c5e0d0c769f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:48.054829 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.054722    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-tls-certs\") pod \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\" (UID: \"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f\") "
Oct 30 19:45:48.054829 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.054819    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-tmp-dir\") pod \"12d39453-861c-437a-8bbe-e9942b8778d3\" (UID: \"12d39453-861c-437a-8bbe-e9942b8778d3\") "
Oct 30 19:45:48.054974 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.054875    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-tmp-dir" pod "4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" (UID: "4d5b13af-b0c4-4343-80ca-5c5e0d0c769f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:48.056115 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.055942    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-model-cache" pod "12d39453-861c-437a-8bbe-e9942b8778d3" (UID: "12d39453-861c-437a-8bbe-e9942b8778d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:48.056115 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.056079    4712 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-home\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:48.056430 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.056323    4712 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-model-cache\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:48.056430 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.056348    4712 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-home\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:48.056430 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.056380    4712 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-model-cache\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:48.056430 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.056394    4712 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-tmp-dir\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:48.056430 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.056149    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-tmp-dir" pod "12d39453-861c-437a-8bbe-e9942b8778d3" (UID: "12d39453-861c-437a-8bbe-e9942b8778d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:48.060325 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.060292    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-tls-certs" pod "4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" (UID: "4d5b13af-b0c4-4343-80ca-5c5e0d0c769f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:48.060763 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.060736    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d39453-861c-437a-8bbe-e9942b8778d3-kube-api-access-srw2b" pod "12d39453-861c-437a-8bbe-e9942b8778d3" (UID: "12d39453-861c-437a-8bbe-e9942b8778d3"). InnerVolumeSpecName "kube-api-access-srw2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:48.060877 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.060849    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-kube-api-access-zs7m6" pod "4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" (UID: "4d5b13af-b0c4-4343-80ca-5c5e0d0c769f"). InnerVolumeSpecName "kube-api-access-zs7m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:48.061274 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.061251    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-dshm" pod "4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" (UID: "4d5b13af-b0c4-4343-80ca-5c5e0d0c769f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:48.061341 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.061295    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12d39453-861c-437a-8bbe-e9942b8778d3-tls-certs" pod "12d39453-861c-437a-8bbe-e9942b8778d3" (UID: "12d39453-861c-437a-8bbe-e9942b8778d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:48.061851 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.061830    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-dshm" pod "12d39453-861c-437a-8bbe-e9942b8778d3" (UID: "12d39453-861c-437a-8bbe-e9942b8778d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:48.163263 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.161212    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs7m6\" (UniqueName: \"kubernetes.io/projected/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-kube-api-access-zs7m6\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:48.163263 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.161275    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srw2b\" (UniqueName: \"kubernetes.io/projected/12d39453-861c-437a-8bbe-e9942b8778d3-kube-api-access-srw2b\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:48.163263 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.161291    4712 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-dshm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:48.163263 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.161304    4712 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f-tls-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:48.163263 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.161320    4712 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-tmp-dir\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:48.163263 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.161333    4712 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12d39453-861c-437a-8bbe-e9942b8778d3-tls-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:48.163263 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.161344    4712 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12d39453-861c-437a-8bbe-e9942b8778d3-dshm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:48.493870 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.493782    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm" event={"ID":"804735e2-421e-40c5-8254-b83c9171ce7c","Type":"ContainerStarted","Data":"140b8df38c6e519f68bd0c780ab509fde97e90c5633edffc4961c22c5dff2a4e"}
Oct 30 19:45:48.493870 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.493851    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm" event={"ID":"804735e2-421e-40c5-8254-b83c9171ce7c","Type":"ContainerStarted","Data":"fe6830394e6548b4d7c0b3be19b466cd264e0a9d43c96859be66045c65c5c3e1"}
Oct 30 19:45:48.495595 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.495561    4712 generic.go:356] "Generic (PLEG): container finished" podID="12d39453-861c-437a-8bbe-e9942b8778d3" containerID="6a99b12677b8af0c56071f1f82b636ef5cdf3d052f0674136ce8ac3cae7dca0f" exitCode=0
Oct 30 19:45:48.495757 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.495711    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" event={"ID":"12d39453-861c-437a-8bbe-e9942b8778d3","Type":"ContainerDied","Data":"6a99b12677b8af0c56071f1f82b636ef5cdf3d052f0674136ce8ac3cae7dca0f"}
Oct 30 19:45:48.495757 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.495740    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj" event={"ID":"12d39453-861c-437a-8bbe-e9942b8778d3","Type":"ContainerDied","Data":"a8b8e952cbed78e359f340935c796bcf29e4f3931400296e79019146961f54f5"}
Oct 30 19:45:48.495871 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.495759    4712 scope.go:122] "RemoveContainer" containerID="6a99b12677b8af0c56071f1f82b636ef5cdf3d052f0674136ce8ac3cae7dca0f"
Oct 30 19:45:48.495871 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.495869    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"
Oct 30 19:45:48.501465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.500842    4712 generic.go:356] "Generic (PLEG): container finished" podID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerID="c7884fb36ae79ee5584ddbe62cedf8720bc522853580c91b186f2dfdd76498ae" exitCode=0
Oct 30 19:45:48.501465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.500907    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" event={"ID":"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f","Type":"ContainerDied","Data":"c7884fb36ae79ee5584ddbe62cedf8720bc522853580c91b186f2dfdd76498ae"}
Oct 30 19:45:48.501465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.500933    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh" event={"ID":"4d5b13af-b0c4-4343-80ca-5c5e0d0c769f","Type":"ContainerDied","Data":"6ac4aa851899c1124e77db67f5aced447f46d7fefe038b963863747e47222893"}
Oct 30 19:45:48.501465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.500995    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"
Oct 30 19:45:48.511334 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.511249    4712 scope.go:122] "RemoveContainer" containerID="6a99b12677b8af0c56071f1f82b636ef5cdf3d052f0674136ce8ac3cae7dca0f"
Oct 30 19:45:48.511747 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:48.511628    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a99b12677b8af0c56071f1f82b636ef5cdf3d052f0674136ce8ac3cae7dca0f\": container with ID starting with 6a99b12677b8af0c56071f1f82b636ef5cdf3d052f0674136ce8ac3cae7dca0f not found: ID does not exist" containerID="6a99b12677b8af0c56071f1f82b636ef5cdf3d052f0674136ce8ac3cae7dca0f"
Oct 30 19:45:48.511747 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.511658    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a99b12677b8af0c56071f1f82b636ef5cdf3d052f0674136ce8ac3cae7dca0f"} err="failed to get container status \"6a99b12677b8af0c56071f1f82b636ef5cdf3d052f0674136ce8ac3cae7dca0f\": rpc error: code = NotFound desc = could not find container \"6a99b12677b8af0c56071f1f82b636ef5cdf3d052f0674136ce8ac3cae7dca0f\": container with ID starting with 6a99b12677b8af0c56071f1f82b636ef5cdf3d052f0674136ce8ac3cae7dca0f not found: ID does not exist"
Oct 30 19:45:48.511747 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.511680    4712 scope.go:122] "RemoveContainer" containerID="c7884fb36ae79ee5584ddbe62cedf8720bc522853580c91b186f2dfdd76498ae"
Oct 30 19:45:48.526818 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.526799    4712 scope.go:122] "RemoveContainer" containerID="c7884fb36ae79ee5584ddbe62cedf8720bc522853580c91b186f2dfdd76498ae"
Oct 30 19:45:48.527090 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:48.527069    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7884fb36ae79ee5584ddbe62cedf8720bc522853580c91b186f2dfdd76498ae\": container with ID starting with c7884fb36ae79ee5584ddbe62cedf8720bc522853580c91b186f2dfdd76498ae not found: ID does not exist" containerID="c7884fb36ae79ee5584ddbe62cedf8720bc522853580c91b186f2dfdd76498ae"
Oct 30 19:45:48.527182 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.527103    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7884fb36ae79ee5584ddbe62cedf8720bc522853580c91b186f2dfdd76498ae"} err="failed to get container status \"c7884fb36ae79ee5584ddbe62cedf8720bc522853580c91b186f2dfdd76498ae\": rpc error: code = NotFound desc = could not find container \"c7884fb36ae79ee5584ddbe62cedf8720bc522853580c91b186f2dfdd76498ae\": container with ID starting with c7884fb36ae79ee5584ddbe62cedf8720bc522853580c91b186f2dfdd76498ae not found: ID does not exist"
Oct 30 19:45:48.550052 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.550014    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm" podStartSLOduration=1.5500013689999999 podStartE2EDuration="1.550001369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:48.549704807 +0000 UTC m=+1501.848966943" watchObservedRunningTime="2026-10-30 19:45:48.550001369 +0000 UTC m=+1501.849263514"
Oct 30 19:45:48.593226 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.593197    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"]
Oct 30 19:45:48.602833 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.602806    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3735b2/simulator-w2-3735b2-kserve-54699dfd4c-mmklj"]
Oct 30 19:45:48.664167 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.664139    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"]
Oct 30 19:45:48.670094 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.670068    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3735b2/distinct-2-w2-3735b2-kserve-7b694fbf46-qvdzh"]
Oct 30 19:45:48.854073 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.853999    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" path="/var/lib/kubelet/pods/12d39453-861c-437a-8bbe-e9942b8778d3/volumes"
Oct 30 19:45:48.854561 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:48.854545    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" path="/var/lib/kubelet/pods/4d5b13af-b0c4-4343-80ca-5c5e0d0c769f/volumes"
Oct 30 19:45:49.514755 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:49.514717    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:49.519417 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:49.519384    4712 generic.go:356] "Generic (PLEG): container finished" podID="804735e2-421e-40c5-8254-b83c9171ce7c" containerID="140b8df38c6e519f68bd0c780ab509fde97e90c5633edffc4961c22c5dff2a4e" exitCode=0
Oct 30 19:45:49.519574 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:49.519492    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm" event={"ID":"804735e2-421e-40c5-8254-b83c9171ce7c","Type":"ContainerDied","Data":"140b8df38c6e519f68bd0c780ab509fde97e90c5633edffc4961c22c5dff2a4e"}
Oct 30 19:45:50.689102 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:50.689078    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm"
Oct 30 19:45:50.793677 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:50.793604    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/804735e2-421e-40c5-8254-b83c9171ce7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804735e2-421e-40c5-8254-b83c9171ce7c-maas-api-openshift-ca\") pod \"804735e2-421e-40c5-8254-b83c9171ce7c\" (UID: \"804735e2-421e-40c5-8254-b83c9171ce7c\") "
Oct 30 19:45:50.793854 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:50.793806    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/804735e2-421e-40c5-8254-b83c9171ce7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804735e2-421e-40c5-8254-b83c9171ce7c-maas-api-xks-ca\") pod \"804735e2-421e-40c5-8254-b83c9171ce7c\" (UID: \"804735e2-421e-40c5-8254-b83c9171ce7c\") "
Oct 30 19:45:50.793932 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:50.793893    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804735e2-421e-40c5-8254-b83c9171ce7c-kube-api-access-rr8q8\" (UniqueName: \"kubernetes.io/projected/804735e2-421e-40c5-8254-b83c9171ce7c-kube-api-access-rr8q8\") pod \"804735e2-421e-40c5-8254-b83c9171ce7c\" (UID: \"804735e2-421e-40c5-8254-b83c9171ce7c\") "
Oct 30 19:45:50.794044 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:50.794019    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804735e2-421e-40c5-8254-b83c9171ce7c-maas-api-openshift-ca" pod "804735e2-421e-40c5-8254-b83c9171ce7c" (UID: "804735e2-421e-40c5-8254-b83c9171ce7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:50.794403 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:50.794382    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804735e2-421e-40c5-8254-b83c9171ce7c-maas-api-openshift-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:50.795948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:50.795925    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/804735e2-421e-40c5-8254-b83c9171ce7c-maas-api-xks-ca" pod "804735e2-421e-40c5-8254-b83c9171ce7c" (UID: "804735e2-421e-40c5-8254-b83c9171ce7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:50.796056 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:50.796013    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804735e2-421e-40c5-8254-b83c9171ce7c-kube-api-access-rr8q8" pod "804735e2-421e-40c5-8254-b83c9171ce7c" (UID: "804735e2-421e-40c5-8254-b83c9171ce7c"). InnerVolumeSpecName "kube-api-access-rr8q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:50.895430 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:50.895405    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804735e2-421e-40c5-8254-b83c9171ce7c-maas-api-xks-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:50.895430 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:50.895431    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr8q8\" (UniqueName: \"kubernetes.io/projected/804735e2-421e-40c5-8254-b83c9171ce7c-kube-api-access-rr8q8\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:51.188171 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:51.188130    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:51.537114 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:51.537044    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm"
Oct 30 19:45:51.537114 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:51.537073    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm" event={"ID":"804735e2-421e-40c5-8254-b83c9171ce7c","Type":"ContainerDied","Data":"fe6830394e6548b4d7c0b3be19b466cd264e0a9d43c96859be66045c65c5c3e1"}
Oct 30 19:45:51.537114 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:51.537112    4712 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe6830394e6548b4d7c0b3be19b466cd264e0a9d43c96859be66045c65c5c3e1"
Oct 30 19:45:51.997744 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:51.997721    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:45:52.109014 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.108935    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4ea703f-3d59-4204-9630-e5915516c074-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ea703f-3d59-4204-9630-e5915516c074-istio-podinfo\") pod \"b4ea703f-3d59-4204-9630-e5915516c074\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") "
Oct 30 19:45:52.109180 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.109022    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ea703f-3d59-4204-9630-e5915516c074-kube-api-access-b94r7\" (UniqueName: \"kubernetes.io/projected/b4ea703f-3d59-4204-9630-e5915516c074-kube-api-access-b94r7\") pod \"b4ea703f-3d59-4204-9630-e5915516c074\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") "
Oct 30 19:45:52.109180 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.109052    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-workload-certs\") pod \"b4ea703f-3d59-4204-9630-e5915516c074\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") "
Oct 30 19:45:52.109180 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.109080    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-istio-envoy\") pod \"b4ea703f-3d59-4204-9630-e5915516c074\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") "
Oct 30 19:45:52.109180 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.109123    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4ea703f-3d59-4204-9630-e5915516c074-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ea703f-3d59-4204-9630-e5915516c074-istiod-ca-cert\") pod \"b4ea703f-3d59-4204-9630-e5915516c074\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") "
Oct 30 19:45:52.109180 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.109160    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ea703f-3d59-4204-9630-e5915516c074-istio-token\" (UniqueName: \"kubernetes.io/projected/b4ea703f-3d59-4204-9630-e5915516c074-istio-token\") pod \"b4ea703f-3d59-4204-9630-e5915516c074\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") "
Oct 30 19:45:52.109528 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.109434    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-workload-certs" pod "b4ea703f-3d59-4204-9630-e5915516c074" (UID: "b4ea703f-3d59-4204-9630-e5915516c074"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:52.109528 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.109451    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-credential-socket\") pod \"b4ea703f-3d59-4204-9630-e5915516c074\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") "
Oct 30 19:45:52.109642 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.109579    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-workload-socket\") pod \"b4ea703f-3d59-4204-9630-e5915516c074\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") "
Oct 30 19:45:52.109642 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.109601    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-istio-data\") pod \"b4ea703f-3d59-4204-9630-e5915516c074\" (UID: \"b4ea703f-3d59-4204-9630-e5915516c074\") "
Oct 30 19:45:52.109753 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.109719    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-credential-socket" pod "b4ea703f-3d59-4204-9630-e5915516c074" (UID: "b4ea703f-3d59-4204-9630-e5915516c074"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:52.109892 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.109864    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ea703f-3d59-4204-9630-e5915516c074-istiod-ca-cert" pod "b4ea703f-3d59-4204-9630-e5915516c074" (UID: "b4ea703f-3d59-4204-9630-e5915516c074"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:52.110039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.110013    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-workload-socket" pod "b4ea703f-3d59-4204-9630-e5915516c074" (UID: "b4ea703f-3d59-4204-9630-e5915516c074"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:52.110119 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.110090    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-istio-data" pod "b4ea703f-3d59-4204-9630-e5915516c074" (UID: "b4ea703f-3d59-4204-9630-e5915516c074"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:52.110471 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.110419    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:52.110572 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.110476    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ea703f-3d59-4204-9630-e5915516c074-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:52.110572 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.110490    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:52.110572 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.110503    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:52.110572 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.110517    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:52.111341 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.111315    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4ea703f-3d59-4204-9630-e5915516c074-istio-podinfo" pod "b4ea703f-3d59-4204-9630-e5915516c074" (UID: "b4ea703f-3d59-4204-9630-e5915516c074"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:45:52.111549 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.111522    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-istio-envoy" pod "b4ea703f-3d59-4204-9630-e5915516c074" (UID: "b4ea703f-3d59-4204-9630-e5915516c074"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:45:52.111617 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.111598    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ea703f-3d59-4204-9630-e5915516c074-istio-token" pod "b4ea703f-3d59-4204-9630-e5915516c074" (UID: "b4ea703f-3d59-4204-9630-e5915516c074"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:52.112094 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.112072    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ea703f-3d59-4204-9630-e5915516c074-kube-api-access-b94r7" pod "b4ea703f-3d59-4204-9630-e5915516c074" (UID: "b4ea703f-3d59-4204-9630-e5915516c074"). InnerVolumeSpecName "kube-api-access-b94r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:52.198776 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.198737    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:52.212671 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.212634    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ea703f-3d59-4204-9630-e5915516c074-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:52.212930 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.212901    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b94r7\" (UniqueName: \"kubernetes.io/projected/b4ea703f-3d59-4204-9630-e5915516c074-kube-api-access-b94r7\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:52.212930 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.212931    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ea703f-3d59-4204-9630-e5915516c074-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:52.213323 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.212946    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ea703f-3d59-4204-9630-e5915516c074-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:45:52.367403 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.367344    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:52.544813 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.544782    4712 generic.go:356] "Generic (PLEG): container finished" podID="b4ea703f-3d59-4204-9630-e5915516c074" containerID="4719c1e72b41ea6049d29de1aef234688f687bd62d0a11c1f6fead039f9d329c" exitCode=0
Oct 30 19:45:52.544996 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.544831    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"
Oct 30 19:45:52.544996 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.544872    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh" event={"ID":"b4ea703f-3d59-4204-9630-e5915516c074","Type":"ContainerDied","Data":"4719c1e72b41ea6049d29de1aef234688f687bd62d0a11c1f6fead039f9d329c"}
Oct 30 19:45:52.544996 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.544961    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh" event={"ID":"b4ea703f-3d59-4204-9630-e5915516c074","Type":"ContainerDied","Data":"ef55a22adfdf02dface9de9758bbff3e8fe332c66ed3b21e25fb22f1140f16ce"}
Oct 30 19:45:52.544996 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.544993    4712 scope.go:122] "RemoveContainer" containerID="4719c1e72b41ea6049d29de1aef234688f687bd62d0a11c1f6fead039f9d329c"
Oct 30 19:45:52.562020 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.561999    4712 scope.go:122] "RemoveContainer" containerID="7abee6196605e7ab09e2cc31da25c149f0da1578809c563848315a264d9b0ed6"
Oct 30 19:45:52.572538 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.572490    4712 scope.go:122] "RemoveContainer" containerID="4719c1e72b41ea6049d29de1aef234688f687bd62d0a11c1f6fead039f9d329c"
Oct 30 19:45:52.573043 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:52.572975    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4719c1e72b41ea6049d29de1aef234688f687bd62d0a11c1f6fead039f9d329c\": container with ID starting with 4719c1e72b41ea6049d29de1aef234688f687bd62d0a11c1f6fead039f9d329c not found: ID does not exist" containerID="4719c1e72b41ea6049d29de1aef234688f687bd62d0a11c1f6fead039f9d329c"
Oct 30 19:45:52.573043 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.573012    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4719c1e72b41ea6049d29de1aef234688f687bd62d0a11c1f6fead039f9d329c"} err="failed to get container status \"4719c1e72b41ea6049d29de1aef234688f687bd62d0a11c1f6fead039f9d329c\": rpc error: code = NotFound desc = could not find container \"4719c1e72b41ea6049d29de1aef234688f687bd62d0a11c1f6fead039f9d329c\": container with ID starting with 4719c1e72b41ea6049d29de1aef234688f687bd62d0a11c1f6fead039f9d329c not found: ID does not exist"
Oct 30 19:45:52.573199 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.573050    4712 scope.go:122] "RemoveContainer" containerID="7abee6196605e7ab09e2cc31da25c149f0da1578809c563848315a264d9b0ed6"
Oct 30 19:45:52.573390 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:45:52.573338    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7abee6196605e7ab09e2cc31da25c149f0da1578809c563848315a264d9b0ed6\": container with ID starting with 7abee6196605e7ab09e2cc31da25c149f0da1578809c563848315a264d9b0ed6 not found: ID does not exist" containerID="7abee6196605e7ab09e2cc31da25c149f0da1578809c563848315a264d9b0ed6"
Oct 30 19:45:52.573465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.573402    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abee6196605e7ab09e2cc31da25c149f0da1578809c563848315a264d9b0ed6"} err="failed to get container status \"7abee6196605e7ab09e2cc31da25c149f0da1578809c563848315a264d9b0ed6\": rpc error: code = NotFound desc = could not find container \"7abee6196605e7ab09e2cc31da25c149f0da1578809c563848315a264d9b0ed6\": container with ID starting with 7abee6196605e7ab09e2cc31da25c149f0da1578809c563848315a264d9b0ed6 not found: ID does not exist"
Oct 30 19:45:52.639437 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.629965    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:52.639611 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.639507    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"]
Oct 30 19:45:52.652392 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.651781    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0c305f-openshift-default-b5dd58b49-z7klh"]
Oct 30 19:45:52.857165 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.857127    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ea703f-3d59-4204-9630-e5915516c074" path="/var/lib/kubelet/pods/b4ea703f-3d59-4204-9630-e5915516c074/volumes"
Oct 30 19:45:52.912535 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:52.912442    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:53.131018 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:53.130983    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:54.746382 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:54.746341    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:56.572578 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:56.572545    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:57.973593 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:57.973558    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:58.081497 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:58.081467    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"]
Oct 30 19:45:58.192268 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:58.192222    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:58.373411 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:58.373375    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:59.463977 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:59.463948    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:45:59.655460 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:45:59.655429    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:00.743891 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:00.743853    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:02.798048 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:02.798014    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:04.137586 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:04.137549    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:04.683902 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:04.683860    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:05.216857 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:05.216816    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:06.374347 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:46:06.374304    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:46:07.226435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:07.226390    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:08.050947 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:08.050917    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:08.271085 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:08.271054    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:09.132728 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:09.132698    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:10.532672 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.532638    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"]
Oct 30 19:46:10.533063 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.532873    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn" podUID="27c8e215-0d99-44f8-88cd-8c5fe3b964b1" containerName="maas-api" containerID="cri-o://78eec3eddcb29273daf501ee57350efeecc3a265b02da12aa6745bdefe8c382d" gracePeriod=30
Oct 30 19:46:10.624836 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.624807    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:10.674054 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.674025    4712 generic.go:356] "Generic (PLEG): container finished" podID="27c8e215-0d99-44f8-88cd-8c5fe3b964b1" containerID="78eec3eddcb29273daf501ee57350efeecc3a265b02da12aa6745bdefe8c382d" exitCode=0
Oct 30 19:46:10.674199 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.674098    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn" event={"ID":"27c8e215-0d99-44f8-88cd-8c5fe3b964b1","Type":"ContainerDied","Data":"78eec3eddcb29273daf501ee57350efeecc3a265b02da12aa6745bdefe8c382d"}
Oct 30 19:46:10.790103 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.790035    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:10.817476 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.817440    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"
Oct 30 19:46:10.869567 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.869534    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-metrics-certs\") pod \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\" (UID: \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\") "
Oct 30 19:46:10.870678 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.870652    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-openshift-service-ca\") pod \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\" (UID: \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\") "
Oct 30 19:46:10.870905 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.870879    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-kube-api-access-r92v8\" (UniqueName: \"kubernetes.io/projected/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-kube-api-access-r92v8\") pod \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\" (UID: \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\") "
Oct 30 19:46:10.871101 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.871086    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-maas-api-tls\") pod \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\" (UID: \"27c8e215-0d99-44f8-88cd-8c5fe3b964b1\") "
Oct 30 19:46:10.892167 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.874718    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-openshift-service-ca" pod "27c8e215-0d99-44f8-88cd-8c5fe3b964b1" (UID: "27c8e215-0d99-44f8-88cd-8c5fe3b964b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:10.892167 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.882100    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-kube-api-access-r92v8" pod "27c8e215-0d99-44f8-88cd-8c5fe3b964b1" (UID: "27c8e215-0d99-44f8-88cd-8c5fe3b964b1"). InnerVolumeSpecName "kube-api-access-r92v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:10.892167 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.884532    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-metrics-certs" pod "27c8e215-0d99-44f8-88cd-8c5fe3b964b1" (UID: "27c8e215-0d99-44f8-88cd-8c5fe3b964b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:10.892167 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.887544    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-maas-api-tls" pod "27c8e215-0d99-44f8-88cd-8c5fe3b964b1" (UID: "27c8e215-0d99-44f8-88cd-8c5fe3b964b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:10.977409 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.973433    4712 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-metrics-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:10.977409 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.973469    4712 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-openshift-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:10.977409 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.973483    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r92v8\" (UniqueName: \"kubernetes.io/projected/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-kube-api-access-r92v8\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:10.977409 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:10.973497    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27c8e215-0d99-44f8-88cd-8c5fe3b964b1-maas-api-tls\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:11.087269 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:11.087195    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:11.381455 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:11.369689    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:11.451679 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:11.451643    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl"]
Oct 30 19:46:11.454184 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:11.454162    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3735b2-hxgbl"]
Oct 30 19:46:11.590972 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:11.590813    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:11.684116 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:11.683995    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn" event={"ID":"27c8e215-0d99-44f8-88cd-8c5fe3b964b1","Type":"ContainerDied","Data":"c3e5bd11fc11152954d27d33d88f3f096f83fa4d37b2ade4188347dbb03caae4"}
Oct 30 19:46:11.684116 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:11.684066    4712 scope.go:122] "RemoveContainer" containerID="78eec3eddcb29273daf501ee57350efeecc3a265b02da12aa6745bdefe8c382d"
Oct 30 19:46:11.684357 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:11.684245    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"
Oct 30 19:46:11.736673 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:11.736647    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"]
Oct 30 19:46:11.749666 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:11.749640    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-3735b2-785767c677-p5qtn"]
Oct 30 19:46:11.907284 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:11.907254    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:11.978727 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:11.978659    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm"]
Oct 30 19:46:11.986688 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:11.986663    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e0e300-5drzm"]
Oct 30 19:46:12.112987 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:12.112948    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:12.566629 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:12.566596    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:12.853200 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:12.853125    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c8e215-0d99-44f8-88cd-8c5fe3b964b1" path="/var/lib/kubelet/pods/27c8e215-0d99-44f8-88cd-8c5fe3b964b1/volumes"
Oct 30 19:46:12.853657 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:12.853640    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="804735e2-421e-40c5-8254-b83c9171ce7c" path="/var/lib/kubelet/pods/804735e2-421e-40c5-8254-b83c9171ce7c/volumes"
Oct 30 19:46:12.854101 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:12.854079    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b431ef98-7f6b-4e01-9cc1-32525af19139" path="/var/lib/kubelet/pods/b431ef98-7f6b-4e01-9cc1-32525af19139/volumes"
Oct 30 19:46:12.951999 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:12.951973    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:13.102700 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.102669    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main" containerID="cri-o://60bc701d79e0d6bffad3395b78027803f54bc7c04a343fbda6448086657b520c" gracePeriod=45
Oct 30 19:46:13.380059 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.380034    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:46:13.518955 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.518922    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-kube-api-access-dpscf\" (UniqueName: \"kubernetes.io/projected/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-kube-api-access-dpscf\") pod \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") "
Oct 30 19:46:13.519139 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.518961    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-model-cache\") pod \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") "
Oct 30 19:46:13.519139 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.519111    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-dshm\") pod \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") "
Oct 30 19:46:13.519252 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.519186    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-home\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-home\") pod \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") "
Oct 30 19:46:13.519252 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.519221    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-tmp-dir\") pod \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") "
Oct 30 19:46:13.519339 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.519278    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-model-cache" pod "5a486ae6-e874-4317-ab9b-e6cdebc7ce89" (UID: "5a486ae6-e874-4317-ab9b-e6cdebc7ce89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:13.519417 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.519399    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-tls-certs\") pod \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\" (UID: \"5a486ae6-e874-4317-ab9b-e6cdebc7ce89\") "
Oct 30 19:46:13.519471 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.519433    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-home" pod "5a486ae6-e874-4317-ab9b-e6cdebc7ce89" (UID: "5a486ae6-e874-4317-ab9b-e6cdebc7ce89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:13.519527 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.519503    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-tmp-dir" pod "5a486ae6-e874-4317-ab9b-e6cdebc7ce89" (UID: "5a486ae6-e874-4317-ab9b-e6cdebc7ce89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:13.520018 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.519980    4712 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-model-cache\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:13.520018 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.520010    4712 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-home\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:13.520240 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.520025    4712 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-tmp-dir\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:13.521590 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.521564    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-kube-api-access-dpscf" pod "5a486ae6-e874-4317-ab9b-e6cdebc7ce89" (UID: "5a486ae6-e874-4317-ab9b-e6cdebc7ce89"). InnerVolumeSpecName "kube-api-access-dpscf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:13.521693 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.521609    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-dshm" pod "5a486ae6-e874-4317-ab9b-e6cdebc7ce89" (UID: "5a486ae6-e874-4317-ab9b-e6cdebc7ce89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:13.521693 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.521640    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-tls-certs" pod "5a486ae6-e874-4317-ab9b-e6cdebc7ce89" (UID: "5a486ae6-e874-4317-ab9b-e6cdebc7ce89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:13.621539 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.621503    4712 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-tls-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:13.621539 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.621530    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpscf\" (UniqueName: \"kubernetes.io/projected/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-kube-api-access-dpscf\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:13.621539 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.621540    4712 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a486ae6-e874-4317-ab9b-e6cdebc7ce89-dshm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:13.699675 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.699634    4712 generic.go:356] "Generic (PLEG): container finished" podID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerID="60bc701d79e0d6bffad3395b78027803f54bc7c04a343fbda6448086657b520c" exitCode=0
Oct 30 19:46:13.699845 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.699671    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" event={"ID":"5a486ae6-e874-4317-ab9b-e6cdebc7ce89","Type":"ContainerDied","Data":"60bc701d79e0d6bffad3395b78027803f54bc7c04a343fbda6448086657b520c"}
Oct 30 19:46:13.699845 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.699716    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj" event={"ID":"5a486ae6-e874-4317-ab9b-e6cdebc7ce89","Type":"ContainerDied","Data":"e84435733914f07e72b02865e077aca4d635ffd27dd3d69fef6a5a67f37321de"}
Oct 30 19:46:13.699845 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.699735    4712 scope.go:122] "RemoveContainer" containerID="60bc701d79e0d6bffad3395b78027803f54bc7c04a343fbda6448086657b520c"
Oct 30 19:46:13.699845 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.699681    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"
Oct 30 19:46:13.709962 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.709946    4712 scope.go:122] "RemoveContainer" containerID="60bc701d79e0d6bffad3395b78027803f54bc7c04a343fbda6448086657b520c"
Oct 30 19:46:13.710258 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:46:13.710236    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60bc701d79e0d6bffad3395b78027803f54bc7c04a343fbda6448086657b520c\": container with ID starting with 60bc701d79e0d6bffad3395b78027803f54bc7c04a343fbda6448086657b520c not found: ID does not exist" containerID="60bc701d79e0d6bffad3395b78027803f54bc7c04a343fbda6448086657b520c"
Oct 30 19:46:13.710355 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.710276    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60bc701d79e0d6bffad3395b78027803f54bc7c04a343fbda6448086657b520c"} err="failed to get container status \"60bc701d79e0d6bffad3395b78027803f54bc7c04a343fbda6448086657b520c\": rpc error: code = NotFound desc = could not find container \"60bc701d79e0d6bffad3395b78027803f54bc7c04a343fbda6448086657b520c\": container with ID starting with 60bc701d79e0d6bffad3395b78027803f54bc7c04a343fbda6448086657b520c not found: ID does not exist"
Oct 30 19:46:13.755441 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.755416    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"]
Oct 30 19:46:13.765040 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.765020    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ef9a5a/distinct-w6-ef9a5a-kserve-797bb9b856-2ndjj"]
Oct 30 19:46:13.908274 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:13.908158    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:14.853239 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:14.853203    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" path="/var/lib/kubelet/pods/5a486ae6-e874-4317-ab9b-e6cdebc7ce89/volumes"
Oct 30 19:46:15.482014 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:15.481975    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:16.998234 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:16.998194    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:17.293468 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:17.293381    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:17.450988 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:17.450960    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"]
Oct 30 19:46:17.451223 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:17.451155    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5" podUID="82a18b97-a6fc-4f33-848b-86bb8ef66fc1" containerName="istio-proxy" containerID="cri-o://b14f96505bb2a9ae7d78b914c27b646e6e155b32e9fa679b7d9de0bd0cf62753" gracePeriod=30
Oct 30 19:46:17.463135 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:17.463102    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"]
Oct 30 19:46:17.463334 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:17.463313    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6" podUID="33d1922a-a6fa-4a8e-8148-174f2ef400af" containerName="istio-proxy" containerID="cri-o://4d8c5a2733261086e6f340825dd861d3ba0a3b0f5d792e9ce68d3648f56fbf55" gracePeriod=30
Oct 30 19:46:17.658041 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:17.658007    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:17.793838 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:17.793805    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"]
Oct 30 19:46:17.794027 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:17.794004    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6" podUID="3d36e5e5-7f21-45a4-961f-61196c8a5e3e" containerName="istio-proxy" containerID="cri-o://a5ef458c28ac36c782dc400e61082a796b4a76ec9b6806c8b2b952a593560559" gracePeriod=30
Oct 30 19:46:19.161679 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:19.161647    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:19.335169 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:19.335136    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:20.759007 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:20.758975    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:20.991707 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:20.991676    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:21.166695 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:21.166664    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:21.344812 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:21.344785    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:21.459481 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:21.459393    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"]
Oct 30 19:46:21.460056 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:21.459725    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v" podUID="7f364e92-b446-4671-8a42-573ebe98d729" containerName="istio-proxy" containerID="cri-o://51408411c28e15e383f2c73f744a85f84b91c5d6ee2ddc58b38d259552566cee" gracePeriod=30
Oct 30 19:46:21.460738 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:21.460300    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"]
Oct 30 19:46:21.460738 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:21.460722    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f" podUID="ced985e1-6978-4dcb-8cc9-7312fe9a16d7" containerName="istio-proxy" containerID="cri-o://f9ee5b84849cb98b147cbc30f4455e3b292ab56a2090571e8a5f345ee05072b2" gracePeriod=30
Oct 30 19:46:21.523457 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:21.523428    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:21.725929 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:21.725848    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:21.915739 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:21.915711    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:22.191606 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.191575    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:22.389755 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.389718    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:22.764060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.763986    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:46:22.791937 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.791907    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:46:22.805439 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.805352    4712 generic.go:356] "Generic (PLEG): container finished" podID="82a18b97-a6fc-4f33-848b-86bb8ef66fc1" containerID="b14f96505bb2a9ae7d78b914c27b646e6e155b32e9fa679b7d9de0bd0cf62753" exitCode=0
Oct 30 19:46:22.805628 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.805611    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5" event={"ID":"82a18b97-a6fc-4f33-848b-86bb8ef66fc1","Type":"ContainerDied","Data":"b14f96505bb2a9ae7d78b914c27b646e6e155b32e9fa679b7d9de0bd0cf62753"}
Oct 30 19:46:22.805719 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.805704    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5" event={"ID":"82a18b97-a6fc-4f33-848b-86bb8ef66fc1","Type":"ContainerDied","Data":"ac10d95d5debaad2606c26fba81f3290507eaafa97ff076d31e6e3e6f3432b42"}
Oct 30 19:46:22.805772 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.805740    4712 scope.go:122] "RemoveContainer" containerID="b14f96505bb2a9ae7d78b914c27b646e6e155b32e9fa679b7d9de0bd0cf62753"
Oct 30 19:46:22.805898 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.805884    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"
Oct 30 19:46:22.810302 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.809976    4712 generic.go:356] "Generic (PLEG): container finished" podID="33d1922a-a6fa-4a8e-8148-174f2ef400af" containerID="4d8c5a2733261086e6f340825dd861d3ba0a3b0f5d792e9ce68d3648f56fbf55" exitCode=0
Oct 30 19:46:22.810302 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.810056    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"
Oct 30 19:46:22.810302 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.810055    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6" event={"ID":"33d1922a-a6fa-4a8e-8148-174f2ef400af","Type":"ContainerDied","Data":"4d8c5a2733261086e6f340825dd861d3ba0a3b0f5d792e9ce68d3648f56fbf55"}
Oct 30 19:46:22.810302 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.810247    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6" event={"ID":"33d1922a-a6fa-4a8e-8148-174f2ef400af","Type":"ContainerDied","Data":"bbff620db2db735864a8d080e1b7673ff67e7c2f106b59f7d266e60c02a61f92"}
Oct 30 19:46:22.833466 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.833435    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:22.868425 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.868403    4712 scope.go:122] "RemoveContainer" containerID="b14f96505bb2a9ae7d78b914c27b646e6e155b32e9fa679b7d9de0bd0cf62753"
Oct 30 19:46:22.868797 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:46:22.868762    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b14f96505bb2a9ae7d78b914c27b646e6e155b32e9fa679b7d9de0bd0cf62753\": container with ID starting with b14f96505bb2a9ae7d78b914c27b646e6e155b32e9fa679b7d9de0bd0cf62753 not found: ID does not exist" containerID="b14f96505bb2a9ae7d78b914c27b646e6e155b32e9fa679b7d9de0bd0cf62753"
Oct 30 19:46:22.868912 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.868798    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b14f96505bb2a9ae7d78b914c27b646e6e155b32e9fa679b7d9de0bd0cf62753"} err="failed to get container status \"b14f96505bb2a9ae7d78b914c27b646e6e155b32e9fa679b7d9de0bd0cf62753\": rpc error: code = NotFound desc = could not find container \"b14f96505bb2a9ae7d78b914c27b646e6e155b32e9fa679b7d9de0bd0cf62753\": container with ID starting with b14f96505bb2a9ae7d78b914c27b646e6e155b32e9fa679b7d9de0bd0cf62753 not found: ID does not exist"
Oct 30 19:46:22.868912 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.868821    4712 scope.go:122] "RemoveContainer" containerID="4d8c5a2733261086e6f340825dd861d3ba0a3b0f5d792e9ce68d3648f56fbf55"
Oct 30 19:46:22.896554 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.896533    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-envoy\") pod \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") "
Oct 30 19:46:22.896666 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.896567    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-workload-certs\") pod \"33d1922a-a6fa-4a8e-8148-174f2ef400af\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") "
Oct 30 19:46:22.896666 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.896648    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-token\" (UniqueName: \"kubernetes.io/projected/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-token\") pod \"33d1922a-a6fa-4a8e-8148-174f2ef400af\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") "
Oct 30 19:46:22.896796 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.896685    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d1922a-a6fa-4a8e-8148-174f2ef400af-kube-api-access-jpkzg\" (UniqueName: \"kubernetes.io/projected/33d1922a-a6fa-4a8e-8148-174f2ef400af-kube-api-access-jpkzg\") pod \"33d1922a-a6fa-4a8e-8148-174f2ef400af\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") "
Oct 30 19:46:22.896796 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.896715    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-token\" (UniqueName: \"kubernetes.io/projected/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-token\") pod \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") "
Oct 30 19:46:22.896796 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.896747    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-data\") pod \"33d1922a-a6fa-4a8e-8148-174f2ef400af\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") "
Oct 30 19:46:22.896796 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.896789    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-credential-socket\") pod \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") "
Oct 30 19:46:22.897004 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.896836    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-workload-certs\") pod \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") "
Oct 30 19:46:22.897004 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.896882    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istiod-ca-cert\") pod \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") "
Oct 30 19:46:22.897004 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.896882    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-workload-certs" pod "33d1922a-a6fa-4a8e-8148-174f2ef400af" (UID: "33d1922a-a6fa-4a8e-8148-174f2ef400af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:22.897004 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.896925    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33d1922a-a6fa-4a8e-8148-174f2ef400af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d1922a-a6fa-4a8e-8148-174f2ef400af-istiod-ca-cert\") pod \"33d1922a-a6fa-4a8e-8148-174f2ef400af\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") "
Oct 30 19:46:22.897004 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.896966    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-workload-socket\") pod \"33d1922a-a6fa-4a8e-8148-174f2ef400af\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") "
Oct 30 19:46:22.897277 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.897017    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-data\") pod \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") "
Oct 30 19:46:22.897277 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.897070    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-workload-socket\") pod \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") "
Oct 30 19:46:22.897277 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.897129    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-envoy\") pod \"33d1922a-a6fa-4a8e-8148-174f2ef400af\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") "
Oct 30 19:46:22.897277 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.897208    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-credential-socket\") pod \"33d1922a-a6fa-4a8e-8148-174f2ef400af\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") "
Oct 30 19:46:22.897277 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.897244    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-podinfo\") pod \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") "
Oct 30 19:46:22.897277 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.897261    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-credential-socket" pod "82a18b97-a6fc-4f33-848b-86bb8ef66fc1" (UID: "82a18b97-a6fc-4f33-848b-86bb8ef66fc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:22.898017 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.897335    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-kube-api-access-gcrx5\" (UniqueName: \"kubernetes.io/projected/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-kube-api-access-gcrx5\") pod \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\" (UID: \"82a18b97-a6fc-4f33-848b-86bb8ef66fc1\") "
Oct 30 19:46:22.898017 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.897392    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-podinfo\") pod \"33d1922a-a6fa-4a8e-8148-174f2ef400af\" (UID: \"33d1922a-a6fa-4a8e-8148-174f2ef400af\") "
Oct 30 19:46:22.898017 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.897463    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-data" pod "33d1922a-a6fa-4a8e-8148-174f2ef400af" (UID: "33d1922a-a6fa-4a8e-8148-174f2ef400af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:22.898017 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.897638    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-workload-certs" pod "82a18b97-a6fc-4f33-848b-86bb8ef66fc1" (UID: "82a18b97-a6fc-4f33-848b-86bb8ef66fc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:22.898757 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.898710    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d1922a-a6fa-4a8e-8148-174f2ef400af-istiod-ca-cert" pod "33d1922a-a6fa-4a8e-8148-174f2ef400af" (UID: "33d1922a-a6fa-4a8e-8148-174f2ef400af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:22.898993 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.898973    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:22.899083 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.898999    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:22.899083 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.899013    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:22.899083 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.899025    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:22.899083 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.899028    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-workload-socket" pod "33d1922a-a6fa-4a8e-8148-174f2ef400af" (UID: "33d1922a-a6fa-4a8e-8148-174f2ef400af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:22.899317 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.899291    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-credential-socket" pod "33d1922a-a6fa-4a8e-8148-174f2ef400af" (UID: "33d1922a-a6fa-4a8e-8148-174f2ef400af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:22.899474 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.899453    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istiod-ca-cert" pod "82a18b97-a6fc-4f33-848b-86bb8ef66fc1" (UID: "82a18b97-a6fc-4f33-848b-86bb8ef66fc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:22.899876 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.899851    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-data" pod "82a18b97-a6fc-4f33-848b-86bb8ef66fc1" (UID: "82a18b97-a6fc-4f33-848b-86bb8ef66fc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:22.900505 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.900481    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-workload-socket" pod "82a18b97-a6fc-4f33-848b-86bb8ef66fc1" (UID: "82a18b97-a6fc-4f33-848b-86bb8ef66fc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:22.901543 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.901498    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-podinfo" pod "33d1922a-a6fa-4a8e-8148-174f2ef400af" (UID: "33d1922a-a6fa-4a8e-8148-174f2ef400af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:22.901543 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.901513    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-envoy" pod "82a18b97-a6fc-4f33-848b-86bb8ef66fc1" (UID: "82a18b97-a6fc-4f33-848b-86bb8ef66fc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:22.902171 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.902147    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-token" pod "82a18b97-a6fc-4f33-848b-86bb8ef66fc1" (UID: "82a18b97-a6fc-4f33-848b-86bb8ef66fc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:22.903891 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.903808    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d1922a-a6fa-4a8e-8148-174f2ef400af-kube-api-access-jpkzg" pod "33d1922a-a6fa-4a8e-8148-174f2ef400af" (UID: "33d1922a-a6fa-4a8e-8148-174f2ef400af"). InnerVolumeSpecName "kube-api-access-jpkzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:22.903992 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.903955    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-token" pod "33d1922a-a6fa-4a8e-8148-174f2ef400af" (UID: "33d1922a-a6fa-4a8e-8148-174f2ef400af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:22.904463 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.904433    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-podinfo" pod "82a18b97-a6fc-4f33-848b-86bb8ef66fc1" (UID: "82a18b97-a6fc-4f33-848b-86bb8ef66fc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:22.905421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.905349    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-envoy" pod "33d1922a-a6fa-4a8e-8148-174f2ef400af" (UID: "33d1922a-a6fa-4a8e-8148-174f2ef400af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:22.905616 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.905585    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-kube-api-access-gcrx5" pod "82a18b97-a6fc-4f33-848b-86bb8ef66fc1" (UID: "82a18b97-a6fc-4f33-848b-86bb8ef66fc1"). InnerVolumeSpecName "kube-api-access-gcrx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:22.908855 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.908832    4712 scope.go:122] "RemoveContainer" containerID="4d8c5a2733261086e6f340825dd861d3ba0a3b0f5d792e9ce68d3648f56fbf55"
Oct 30 19:46:22.909189 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:46:22.909163    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d8c5a2733261086e6f340825dd861d3ba0a3b0f5d792e9ce68d3648f56fbf55\": container with ID starting with 4d8c5a2733261086e6f340825dd861d3ba0a3b0f5d792e9ce68d3648f56fbf55 not found: ID does not exist" containerID="4d8c5a2733261086e6f340825dd861d3ba0a3b0f5d792e9ce68d3648f56fbf55"
Oct 30 19:46:22.909291 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:22.909196    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d8c5a2733261086e6f340825dd861d3ba0a3b0f5d792e9ce68d3648f56fbf55"} err="failed to get container status \"4d8c5a2733261086e6f340825dd861d3ba0a3b0f5d792e9ce68d3648f56fbf55\": rpc error: code = NotFound desc = could not find container \"4d8c5a2733261086e6f340825dd861d3ba0a3b0f5d792e9ce68d3648f56fbf55\": container with ID starting with 4d8c5a2733261086e6f340825dd861d3ba0a3b0f5d792e9ce68d3648f56fbf55 not found: ID does not exist"
Oct 30 19:46:23.001151 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.001068    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.001151 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.001100    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.001151 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.001110    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.001151 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.001120    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcrx5\" (UniqueName: \"kubernetes.io/projected/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-kube-api-access-gcrx5\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.001151 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.001130    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.001151 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.001138    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.001151 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.001147    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33d1922a-a6fa-4a8e-8148-174f2ef400af-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.001151 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.001155    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpkzg\" (UniqueName: \"kubernetes.io/projected/33d1922a-a6fa-4a8e-8148-174f2ef400af-kube-api-access-jpkzg\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.002020 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.001163    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.002020 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.001171    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.002020 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.001181    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33d1922a-a6fa-4a8e-8148-174f2ef400af-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.002020 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.001192    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33d1922a-a6fa-4a8e-8148-174f2ef400af-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.002020 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.001200    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.002020 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.001208    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82a18b97-a6fc-4f33-848b-86bb8ef66fc1-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.050863 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.050840    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:46:23.102679 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.102651    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-workload-certs\") pod \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") "
Oct 30 19:46:23.102815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.102690    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istiod-ca-cert\") pod \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") "
Oct 30 19:46:23.102815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.102749    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-workload-socket\") pod \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") "
Oct 30 19:46:23.102909 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.102845    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-kube-api-access-vd5jv\" (UniqueName: \"kubernetes.io/projected/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-kube-api-access-vd5jv\") pod \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") "
Oct 30 19:46:23.102909 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.102888    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-credential-socket\") pod \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") "
Oct 30 19:46:23.103016 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.102928    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-workload-certs" pod "3d36e5e5-7f21-45a4-961f-61196c8a5e3e" (UID: "3d36e5e5-7f21-45a4-961f-61196c8a5e3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:23.103016 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.102951    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-podinfo\") pod \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") "
Oct 30 19:46:23.103016 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.102999    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-data\") pod \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") "
Oct 30 19:46:23.103185 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.103047    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-envoy\") pod \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") "
Oct 30 19:46:23.103185 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.103099    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-workload-socket" pod "3d36e5e5-7f21-45a4-961f-61196c8a5e3e" (UID: "3d36e5e5-7f21-45a4-961f-61196c8a5e3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:23.103185 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.103127    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istiod-ca-cert" pod "3d36e5e5-7f21-45a4-961f-61196c8a5e3e" (UID: "3d36e5e5-7f21-45a4-961f-61196c8a5e3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:23.103185 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.103142    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-token\" (UniqueName: \"kubernetes.io/projected/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-token\") pod \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\" (UID: \"3d36e5e5-7f21-45a4-961f-61196c8a5e3e\") "
Oct 30 19:46:23.106421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.103780    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.106421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.103817    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.106421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.103830    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.106421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.104088    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-data" pod "3d36e5e5-7f21-45a4-961f-61196c8a5e3e" (UID: "3d36e5e5-7f21-45a4-961f-61196c8a5e3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:23.106421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.104116    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-credential-socket" pod "3d36e5e5-7f21-45a4-961f-61196c8a5e3e" (UID: "3d36e5e5-7f21-45a4-961f-61196c8a5e3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:23.106421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.105926    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-kube-api-access-vd5jv" pod "3d36e5e5-7f21-45a4-961f-61196c8a5e3e" (UID: "3d36e5e5-7f21-45a4-961f-61196c8a5e3e"). InnerVolumeSpecName "kube-api-access-vd5jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:23.106421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.105959    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-podinfo" pod "3d36e5e5-7f21-45a4-961f-61196c8a5e3e" (UID: "3d36e5e5-7f21-45a4-961f-61196c8a5e3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:23.106421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.106051    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-token" pod "3d36e5e5-7f21-45a4-961f-61196c8a5e3e" (UID: "3d36e5e5-7f21-45a4-961f-61196c8a5e3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:23.107313 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.107287    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-envoy" pod "3d36e5e5-7f21-45a4-961f-61196c8a5e3e" (UID: "3d36e5e5-7f21-45a4-961f-61196c8a5e3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:23.154015 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.153988    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"]
Oct 30 19:46:23.158467 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.158447    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-tbsl5"]
Oct 30 19:46:23.194197 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.194172    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"]
Oct 30 19:46:23.198227 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.198206    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-ns8s6"]
Oct 30 19:46:23.205209 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.205180    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.205209 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.205205    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.205335 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.205220    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd5jv\" (UniqueName: \"kubernetes.io/projected/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-kube-api-access-vd5jv\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.205335 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.205233    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.205335 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.205247    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.205335 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.205259    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d36e5e5-7f21-45a4-961f-61196c8a5e3e-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:23.326355 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.326297    4712 patch_prober.go:29] interesting pod/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:46:23.326355 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.326341    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f" podUID="ced985e1-6978-4dcb-8cc9-7312fe9a16d7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:46:23.818790 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.818755    4712 generic.go:356] "Generic (PLEG): container finished" podID="3d36e5e5-7f21-45a4-961f-61196c8a5e3e" containerID="a5ef458c28ac36c782dc400e61082a796b4a76ec9b6806c8b2b952a593560559" exitCode=0
Oct 30 19:46:23.818967 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.818801    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"
Oct 30 19:46:23.818967 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.818835    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6" event={"ID":"3d36e5e5-7f21-45a4-961f-61196c8a5e3e","Type":"ContainerDied","Data":"a5ef458c28ac36c782dc400e61082a796b4a76ec9b6806c8b2b952a593560559"}
Oct 30 19:46:23.818967 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.818894    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6" event={"ID":"3d36e5e5-7f21-45a4-961f-61196c8a5e3e","Type":"ContainerDied","Data":"fe86dedc3e50b1e9ff569b8d2739c0b9f8d5351b2f372490fdeb76116f7c72dc"}
Oct 30 19:46:23.818967 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.818923    4712 scope.go:122] "RemoveContainer" containerID="a5ef458c28ac36c782dc400e61082a796b4a76ec9b6806c8b2b952a593560559"
Oct 30 19:46:23.830479 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.830459    4712 scope.go:122] "RemoveContainer" containerID="a5ef458c28ac36c782dc400e61082a796b4a76ec9b6806c8b2b952a593560559"
Oct 30 19:46:23.830740 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:46:23.830720    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5ef458c28ac36c782dc400e61082a796b4a76ec9b6806c8b2b952a593560559\": container with ID starting with a5ef458c28ac36c782dc400e61082a796b4a76ec9b6806c8b2b952a593560559 not found: ID does not exist" containerID="a5ef458c28ac36c782dc400e61082a796b4a76ec9b6806c8b2b952a593560559"
Oct 30 19:46:23.830786 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.830748    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5ef458c28ac36c782dc400e61082a796b4a76ec9b6806c8b2b952a593560559"} err="failed to get container status \"a5ef458c28ac36c782dc400e61082a796b4a76ec9b6806c8b2b952a593560559\": rpc error: code = NotFound desc = could not find container \"a5ef458c28ac36c782dc400e61082a796b4a76ec9b6806c8b2b952a593560559\": container with ID starting with a5ef458c28ac36c782dc400e61082a796b4a76ec9b6806c8b2b952a593560559 not found: ID does not exist"
Oct 30 19:46:23.871602 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.871576    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"]
Oct 30 19:46:23.876065 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:23.876039    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-h45z6"]
Oct 30 19:46:24.093062 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:24.092988    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:24.852650 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:24.852612    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33d1922a-a6fa-4a8e-8148-174f2ef400af" path="/var/lib/kubelet/pods/33d1922a-a6fa-4a8e-8148-174f2ef400af/volumes"
Oct 30 19:46:24.853117 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:24.853102    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d36e5e5-7f21-45a4-961f-61196c8a5e3e" path="/var/lib/kubelet/pods/3d36e5e5-7f21-45a4-961f-61196c8a5e3e/volumes"
Oct 30 19:46:24.853539 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:24.853526    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82a18b97-a6fc-4f33-848b-86bb8ef66fc1" path="/var/lib/kubelet/pods/82a18b97-a6fc-4f33-848b-86bb8ef66fc1/volumes"
Oct 30 19:46:25.680454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:25.680415    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:26.743356 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.742721    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:46:26.782928 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.782906    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:46:26.843197 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.843161    4712 generic.go:356] "Generic (PLEG): container finished" podID="7f364e92-b446-4671-8a42-573ebe98d729" containerID="51408411c28e15e383f2c73f744a85f84b91c5d6ee2ddc58b38d259552566cee" exitCode=0
Oct 30 19:46:26.843353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.843207    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"
Oct 30 19:46:26.843353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.843241    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v" event={"ID":"7f364e92-b446-4671-8a42-573ebe98d729","Type":"ContainerDied","Data":"51408411c28e15e383f2c73f744a85f84b91c5d6ee2ddc58b38d259552566cee"}
Oct 30 19:46:26.843353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.843288    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v" event={"ID":"7f364e92-b446-4671-8a42-573ebe98d729","Type":"ContainerDied","Data":"852a67726a16c00b5559b85612e55587435b2700b9d3187f09019b4dd496cce5"}
Oct 30 19:46:26.843353 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.843315    4712 scope.go:122] "RemoveContainer" containerID="51408411c28e15e383f2c73f744a85f84b91c5d6ee2ddc58b38d259552566cee"
Oct 30 19:46:26.844837 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.844816    4712 generic.go:356] "Generic (PLEG): container finished" podID="ced985e1-6978-4dcb-8cc9-7312fe9a16d7" containerID="f9ee5b84849cb98b147cbc30f4455e3b292ab56a2090571e8a5f345ee05072b2" exitCode=0
Oct 30 19:46:26.845099 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.845078    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"
Oct 30 19:46:26.847660 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.847635    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-workload-socket\") pod \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") "
Oct 30 19:46:26.847768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.847696    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f364e92-b446-4671-8a42-573ebe98d729-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f364e92-b446-4671-8a42-573ebe98d729-istio-podinfo\") pod \"7f364e92-b446-4671-8a42-573ebe98d729\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") "
Oct 30 19:46:26.847768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.847722    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-credential-socket\") pod \"7f364e92-b446-4671-8a42-573ebe98d729\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") "
Oct 30 19:46:26.847768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.847739    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-workload-socket\") pod \"7f364e92-b446-4671-8a42-573ebe98d729\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") "
Oct 30 19:46:26.848000 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.847980    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-token\" (UniqueName: \"kubernetes.io/projected/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-token\") pod \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") "
Oct 30 19:46:26.848062 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.848037    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f364e92-b446-4671-8a42-573ebe98d729-kube-api-access-xsbg2\" (UniqueName: \"kubernetes.io/projected/7f364e92-b446-4671-8a42-573ebe98d729-kube-api-access-xsbg2\") pod \"7f364e92-b446-4671-8a42-573ebe98d729\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") "
Oct 30 19:46:26.848116 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.848078    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-workload-certs\") pod \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") "
Oct 30 19:46:26.848116 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.848109    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-data\") pod \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") "
Oct 30 19:46:26.848208 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.848142    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-podinfo\") pod \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") "
Oct 30 19:46:26.848208 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.848186    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-credential-socket\") pod \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") "
Oct 30 19:46:26.848329 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.848250    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-istio-envoy\") pod \"7f364e92-b446-4671-8a42-573ebe98d729\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") "
Oct 30 19:46:26.848599 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.848454    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-kube-api-access-cbk8s\" (UniqueName: \"kubernetes.io/projected/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-kube-api-access-cbk8s\") pod \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") "
Oct 30 19:46:26.848599 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.848494    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-workload-certs\") pod \"7f364e92-b446-4671-8a42-573ebe98d729\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") "
Oct 30 19:46:26.848599 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.848572    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-istio-data\") pod \"7f364e92-b446-4671-8a42-573ebe98d729\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") "
Oct 30 19:46:26.848842 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.848666    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istiod-ca-cert\") pod \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") "
Oct 30 19:46:26.848842 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.848751    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f364e92-b446-4671-8a42-573ebe98d729-istio-token\" (UniqueName: \"kubernetes.io/projected/7f364e92-b446-4671-8a42-573ebe98d729-istio-token\") pod \"7f364e92-b446-4671-8a42-573ebe98d729\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") "
Oct 30 19:46:26.848842 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.848808    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-envoy\") pod \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\" (UID: \"ced985e1-6978-4dcb-8cc9-7312fe9a16d7\") "
Oct 30 19:46:26.849000 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.848853    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f364e92-b446-4671-8a42-573ebe98d729-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f364e92-b446-4671-8a42-573ebe98d729-istiod-ca-cert\") pod \"7f364e92-b446-4671-8a42-573ebe98d729\" (UID: \"7f364e92-b446-4671-8a42-573ebe98d729\") "
Oct 30 19:46:26.849152 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.849119    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-credential-socket" pod "7f364e92-b446-4671-8a42-573ebe98d729" (UID: "7f364e92-b446-4671-8a42-573ebe98d729"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.849411 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.849386    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-workload-socket" pod "7f364e92-b446-4671-8a42-573ebe98d729" (UID: "7f364e92-b446-4671-8a42-573ebe98d729"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.849508 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.849455    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f364e92-b446-4671-8a42-573ebe98d729-istiod-ca-cert" pod "7f364e92-b446-4671-8a42-573ebe98d729" (UID: "7f364e92-b446-4671-8a42-573ebe98d729"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:26.849937 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.849707    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f364e92-b446-4671-8a42-573ebe98d729-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:26.849937 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.849733    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:26.849937 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.849751    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:26.849937 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.849733    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-credential-socket" pod "ced985e1-6978-4dcb-8cc9-7312fe9a16d7" (UID: "ced985e1-6978-4dcb-8cc9-7312fe9a16d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.849937 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.849920    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-data" pod "ced985e1-6978-4dcb-8cc9-7312fe9a16d7" (UID: "ced985e1-6978-4dcb-8cc9-7312fe9a16d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.850569 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.850298    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-workload-certs" pod "ced985e1-6978-4dcb-8cc9-7312fe9a16d7" (UID: "ced985e1-6978-4dcb-8cc9-7312fe9a16d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.850974 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.850708    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-istio-data" pod "7f364e92-b446-4671-8a42-573ebe98d729" (UID: "7f364e92-b446-4671-8a42-573ebe98d729"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.852677 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.852651    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istiod-ca-cert" pod "ced985e1-6978-4dcb-8cc9-7312fe9a16d7" (UID: "ced985e1-6978-4dcb-8cc9-7312fe9a16d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:26.852787 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.852759    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f364e92-b446-4671-8a42-573ebe98d729-istio-token" pod "7f364e92-b446-4671-8a42-573ebe98d729" (UID: "7f364e92-b446-4671-8a42-573ebe98d729"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:26.853044 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.853020    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-workload-certs" pod "7f364e92-b446-4671-8a42-573ebe98d729" (UID: "7f364e92-b446-4671-8a42-573ebe98d729"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.853403 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.853320    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-workload-socket" pod "ced985e1-6978-4dcb-8cc9-7312fe9a16d7" (UID: "ced985e1-6978-4dcb-8cc9-7312fe9a16d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.855514 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.855485    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-istio-envoy" pod "7f364e92-b446-4671-8a42-573ebe98d729" (UID: "7f364e92-b446-4671-8a42-573ebe98d729"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.855633 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.855507    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-envoy" pod "ced985e1-6978-4dcb-8cc9-7312fe9a16d7" (UID: "ced985e1-6978-4dcb-8cc9-7312fe9a16d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:26.855633 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.855519    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f364e92-b446-4671-8a42-573ebe98d729-kube-api-access-xsbg2" pod "7f364e92-b446-4671-8a42-573ebe98d729" (UID: "7f364e92-b446-4671-8a42-573ebe98d729"). InnerVolumeSpecName "kube-api-access-xsbg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:26.857860 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.857832    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-podinfo" pod "ced985e1-6978-4dcb-8cc9-7312fe9a16d7" (UID: "ced985e1-6978-4dcb-8cc9-7312fe9a16d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:26.860499 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.859022    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f364e92-b446-4671-8a42-573ebe98d729-istio-podinfo" pod "7f364e92-b446-4671-8a42-573ebe98d729" (UID: "7f364e92-b446-4671-8a42-573ebe98d729"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:26.860499 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.859055    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-token" pod "ced985e1-6978-4dcb-8cc9-7312fe9a16d7" (UID: "ced985e1-6978-4dcb-8cc9-7312fe9a16d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:26.860499 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.860464    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-kube-api-access-cbk8s" pod "ced985e1-6978-4dcb-8cc9-7312fe9a16d7" (UID: "ced985e1-6978-4dcb-8cc9-7312fe9a16d7"). InnerVolumeSpecName "kube-api-access-cbk8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:26.862507 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.862292    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f" event={"ID":"ced985e1-6978-4dcb-8cc9-7312fe9a16d7","Type":"ContainerDied","Data":"f9ee5b84849cb98b147cbc30f4455e3b292ab56a2090571e8a5f345ee05072b2"}
Oct 30 19:46:26.862507 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.862345    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f" event={"ID":"ced985e1-6978-4dcb-8cc9-7312fe9a16d7","Type":"ContainerDied","Data":"011134bd88cc580852104fbd7544641098b416fd0013888ea4f4e3f684043bbe"}
Oct 30 19:46:26.865226 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.865204    4712 scope.go:122] "RemoveContainer" containerID="51408411c28e15e383f2c73f744a85f84b91c5d6ee2ddc58b38d259552566cee"
Oct 30 19:46:26.867497 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:46:26.867345    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51408411c28e15e383f2c73f744a85f84b91c5d6ee2ddc58b38d259552566cee\": container with ID starting with 51408411c28e15e383f2c73f744a85f84b91c5d6ee2ddc58b38d259552566cee not found: ID does not exist" containerID="51408411c28e15e383f2c73f744a85f84b91c5d6ee2ddc58b38d259552566cee"
Oct 30 19:46:26.867497 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.867394    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51408411c28e15e383f2c73f744a85f84b91c5d6ee2ddc58b38d259552566cee"} err="failed to get container status \"51408411c28e15e383f2c73f744a85f84b91c5d6ee2ddc58b38d259552566cee\": rpc error: code = NotFound desc = could not find container \"51408411c28e15e383f2c73f744a85f84b91c5d6ee2ddc58b38d259552566cee\": container with ID starting with 51408411c28e15e383f2c73f744a85f84b91c5d6ee2ddc58b38d259552566cee not found: ID does not exist"
Oct 30 19:46:26.867497 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.867419    4712 scope.go:122] "RemoveContainer" containerID="f9ee5b84849cb98b147cbc30f4455e3b292ab56a2090571e8a5f345ee05072b2"
Oct 30 19:46:26.882619 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.882601    4712 scope.go:122] "RemoveContainer" containerID="f9ee5b84849cb98b147cbc30f4455e3b292ab56a2090571e8a5f345ee05072b2"
Oct 30 19:46:26.882899 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:46:26.882872    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ee5b84849cb98b147cbc30f4455e3b292ab56a2090571e8a5f345ee05072b2\": container with ID starting with f9ee5b84849cb98b147cbc30f4455e3b292ab56a2090571e8a5f345ee05072b2 not found: ID does not exist" containerID="f9ee5b84849cb98b147cbc30f4455e3b292ab56a2090571e8a5f345ee05072b2"
Oct 30 19:46:26.882960 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.882910    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ee5b84849cb98b147cbc30f4455e3b292ab56a2090571e8a5f345ee05072b2"} err="failed to get container status \"f9ee5b84849cb98b147cbc30f4455e3b292ab56a2090571e8a5f345ee05072b2\": rpc error: code = NotFound desc = could not find container \"f9ee5b84849cb98b147cbc30f4455e3b292ab56a2090571e8a5f345ee05072b2\": container with ID starting with f9ee5b84849cb98b147cbc30f4455e3b292ab56a2090571e8a5f345ee05072b2 not found: ID does not exist"
Oct 30 19:46:26.884463 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.884443    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"]
Oct 30 19:46:26.906196 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.906166    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"]
Oct 30 19:46:26.943663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.943635    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"]
Oct 30 19:46:26.951857 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.951832    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:26.951857 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.951859    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f364e92-b446-4671-8a42-573ebe98d729-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:26.951857 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.951872    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:26.952114 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.951886    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:26.952114 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.951899    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f364e92-b446-4671-8a42-573ebe98d729-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:26.952114 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.951913    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:26.952114 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.951926    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsbg2\" (UniqueName: \"kubernetes.io/projected/7f364e92-b446-4671-8a42-573ebe98d729-kube-api-access-xsbg2\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:26.952114 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.951942    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:26.952114 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.951954    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:26.952114 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.951966    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:26.952114 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.951978    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:26.952114 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.951991    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:26.952114 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.952004    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbk8s\" (UniqueName: \"kubernetes.io/projected/ced985e1-6978-4dcb-8cc9-7312fe9a16d7-kube-api-access-cbk8s\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:26.952114 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.952017    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:26.952114 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:26.952031    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f364e92-b446-4671-8a42-573ebe98d729-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:27.180310 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:27.180280    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:27.199915 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:27.199885    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"]
Oct 30 19:46:27.205134 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:27.205109    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-c6r6f"]
Oct 30 19:46:27.245015 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:27.244985    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"]
Oct 30 19:46:27.252453 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:27.252429    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3735b2-openshift-default-69bc97d6f-pgk4v"]
Oct 30 19:46:27.357580 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:27.357546    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:28.852118 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:28.852084    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f364e92-b446-4671-8a42-573ebe98d729" path="/var/lib/kubelet/pods/7f364e92-b446-4671-8a42-573ebe98d729/volumes"
Oct 30 19:46:28.852779 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:28.852747    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced985e1-6978-4dcb-8cc9-7312fe9a16d7" path="/var/lib/kubelet/pods/ced985e1-6978-4dcb-8cc9-7312fe9a16d7/volumes"
Oct 30 19:46:29.354058 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:29.354023    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:30.846869 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:30.846835    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:31.162268 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:31.162236    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:31.475466 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:31.475379    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:33.121545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:33.121515    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:33.286117 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:33.286081    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:34.901801 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:34.901764    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:36.374118 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:46:36.374075    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:46:36.710340 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:36.710213    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:37.202689 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:37.202654    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:37.422919 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:37.422878    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:37.836698 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:37.836666    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:38.101805 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:38.101712    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:38.475199 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:38.475163    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:38.624654 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:38.624620    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:39.804605 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:39.804574    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"]
Oct 30 19:46:39.805086 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:39.804810    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq" podUID="b3cd29f0-6462-40b1-a5a0-30deec5e9321" containerName="maas-api" containerID="cri-o://c56e87437a99acd7ea3b523aeaa619c1a690cc570f243443c4f91ade22ef282f" gracePeriod=30
Oct 30 19:46:39.905656 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:39.905594    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:39.940863 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:39.940836    4712 generic.go:356] "Generic (PLEG): container finished" podID="b3cd29f0-6462-40b1-a5a0-30deec5e9321" containerID="c56e87437a99acd7ea3b523aeaa619c1a690cc570f243443c4f91ade22ef282f" exitCode=0
Oct 30 19:46:39.940990 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:39.940873    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq" event={"ID":"b3cd29f0-6462-40b1-a5a0-30deec5e9321","Type":"ContainerDied","Data":"c56e87437a99acd7ea3b523aeaa619c1a690cc570f243443c4f91ade22ef282f"}
Oct 30 19:46:40.177408 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:40.176873    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:46:40.263383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:40.262773    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3cd29f0-6462-40b1-a5a0-30deec5e9321-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cd29f0-6462-40b1-a5a0-30deec5e9321-metrics-certs\") pod \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\" (UID: \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\") "
Oct 30 19:46:40.263383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:40.262942    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3cd29f0-6462-40b1-a5a0-30deec5e9321-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cd29f0-6462-40b1-a5a0-30deec5e9321-openshift-service-ca\") pod \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\" (UID: \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\") "
Oct 30 19:46:40.263383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:40.262975    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3cd29f0-6462-40b1-a5a0-30deec5e9321-kube-api-access-9jhl5\" (UniqueName: \"kubernetes.io/projected/b3cd29f0-6462-40b1-a5a0-30deec5e9321-kube-api-access-9jhl5\") pod \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\" (UID: \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\") "
Oct 30 19:46:40.263383 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:40.263073    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3cd29f0-6462-40b1-a5a0-30deec5e9321-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cd29f0-6462-40b1-a5a0-30deec5e9321-maas-api-tls\") pod \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\" (UID: \"b3cd29f0-6462-40b1-a5a0-30deec5e9321\") "
Oct 30 19:46:40.277894 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:40.276880    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3cd29f0-6462-40b1-a5a0-30deec5e9321-maas-api-tls" pod "b3cd29f0-6462-40b1-a5a0-30deec5e9321" (UID: "b3cd29f0-6462-40b1-a5a0-30deec5e9321"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:40.277894 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:40.277242    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3cd29f0-6462-40b1-a5a0-30deec5e9321-openshift-service-ca" pod "b3cd29f0-6462-40b1-a5a0-30deec5e9321" (UID: "b3cd29f0-6462-40b1-a5a0-30deec5e9321"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:40.281901 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:40.281003    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cd29f0-6462-40b1-a5a0-30deec5e9321-kube-api-access-9jhl5" pod "b3cd29f0-6462-40b1-a5a0-30deec5e9321" (UID: "b3cd29f0-6462-40b1-a5a0-30deec5e9321"). InnerVolumeSpecName "kube-api-access-9jhl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:40.281901 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:40.281080    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3cd29f0-6462-40b1-a5a0-30deec5e9321-metrics-certs" pod "b3cd29f0-6462-40b1-a5a0-30deec5e9321" (UID: "b3cd29f0-6462-40b1-a5a0-30deec5e9321"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:40.378129 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:40.378052    4712 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3cd29f0-6462-40b1-a5a0-30deec5e9321-metrics-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:40.378129 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:40.378084    4712 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cd29f0-6462-40b1-a5a0-30deec5e9321-openshift-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:40.378129 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:40.378101    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jhl5\" (UniqueName: \"kubernetes.io/projected/b3cd29f0-6462-40b1-a5a0-30deec5e9321-kube-api-access-9jhl5\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:40.378129 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:40.378114    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cd29f0-6462-40b1-a5a0-30deec5e9321-maas-api-tls\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:40.424255 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:40.424193    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:40.591162 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:40.591124    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:40.948153 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:40.948124    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"
Oct 30 19:46:40.948585 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:40.948147    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq" event={"ID":"b3cd29f0-6462-40b1-a5a0-30deec5e9321","Type":"ContainerDied","Data":"1ef67f20943594828f6a7bc330b314abf28028603e7e880fa792f7ab6aa659d3"}
Oct 30 19:46:40.948585 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:40.948209    4712 scope.go:122] "RemoveContainer" containerID="c56e87437a99acd7ea3b523aeaa619c1a690cc570f243443c4f91ade22ef282f"
Oct 30 19:46:41.020120 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:41.020052    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"]
Oct 30 19:46:41.034392 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:41.034317    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-ef9a5a-65c569c756-45xnq"]
Oct 30 19:46:41.174855 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:41.174825    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:41.907328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:41.907265    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" containerID="cri-o://463c5b2a01fe2bb2d5910e2addaf577df50697445bbbdeabe50ed4b2dac3a2d6" gracePeriod=45
Oct 30 19:46:41.926586 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:41.926555    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main" containerID="cri-o://780f333b11a7f7aec3ab0b2d320191cf3a68df4a8a6b408cc3cc19110fa4ed04" gracePeriod=45
Oct 30 19:46:41.970055 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:41.970024    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" containerID="cri-o://b715774721260001c1debf173fc547a728f359b9dc8ac5b2465eb7ffeb926069" gracePeriod=45
Oct 30 19:46:42.015889 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.015856    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": dial tcp 10.133.0.58:8000: connect: connection refused"
Oct 30 19:46:42.025346 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.025319    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": dial tcp 10.133.0.48:8000: connect: connection refused"
Oct 30 19:46:42.259426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.259397    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:46:42.265217 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.265147    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:46:42.273192 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.273170    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:46:42.382505 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.382473    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:42.408897 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.408867    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a3b2eb-8722-4782-af07-70239b3c0ff5-kube-api-access-l499v\" (UniqueName: \"kubernetes.io/projected/d1a3b2eb-8722-4782-af07-70239b3c0ff5-kube-api-access-l499v\") pod \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") "
Oct 30 19:46:42.409091 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.408923    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-dshm\") pod \"aa037781-1c62-492d-adcc-006600af37ff\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") "
Oct 30 19:46:42.409091 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.408970    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-model-cache\") pod \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") "
Oct 30 19:46:42.409091 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409024    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-tls-certs\") pod \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") "
Oct 30 19:46:42.409287 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409105    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-home\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-home\") pod \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") "
Oct 30 19:46:42.409287 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409138    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-model-cache\") pod \"aa037781-1c62-492d-adcc-006600af37ff\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") "
Oct 30 19:46:42.409287 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409185    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa037781-1c62-492d-adcc-006600af37ff-kube-api-access-kknxw\" (UniqueName: \"kubernetes.io/projected/aa037781-1c62-492d-adcc-006600af37ff-kube-api-access-kknxw\") pod \"aa037781-1c62-492d-adcc-006600af37ff\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") "
Oct 30 19:46:42.409287 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409264    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-dshm\") pod \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") "
Oct 30 19:46:42.409542 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409299    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-model-cache\") pod \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") "
Oct 30 19:46:42.409542 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409306    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-model-cache" pod "d1a3b2eb-8722-4782-af07-70239b3c0ff5" (UID: "d1a3b2eb-8722-4782-af07-70239b3c0ff5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:42.409542 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409329    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-tmp-dir\") pod \"aa037781-1c62-492d-adcc-006600af37ff\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") "
Oct 30 19:46:42.409542 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409411    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa037781-1c62-492d-adcc-006600af37ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa037781-1c62-492d-adcc-006600af37ff-tls-certs\") pod \"aa037781-1c62-492d-adcc-006600af37ff\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") "
Oct 30 19:46:42.409542 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409456    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1a3b2eb-8722-4782-af07-70239b3c0ff5-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a3b2eb-8722-4782-af07-70239b3c0ff5-tls-certs\") pod \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") "
Oct 30 19:46:42.409542 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409495    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-dshm\") pod \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") "
Oct 30 19:46:42.409542 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409522    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-tmp-dir\") pod \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") "
Oct 30 19:46:42.409907 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409561    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-kube-api-access-jdqm8\" (UniqueName: \"kubernetes.io/projected/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-kube-api-access-jdqm8\") pod \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") "
Oct 30 19:46:42.409907 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409563    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-model-cache" pod "aa037781-1c62-492d-adcc-006600af37ff" (UID: "aa037781-1c62-492d-adcc-006600af37ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:42.409907 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409571    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-model-cache" pod "cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" (UID: "cdc58f78-58ef-4981-b1a9-4c6aa8b781fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:42.409907 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409598    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-home\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-home\") pod \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\" (UID: \"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc\") "
Oct 30 19:46:42.409907 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409663    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-home\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-home\") pod \"aa037781-1c62-492d-adcc-006600af37ff\" (UID: \"aa037781-1c62-492d-adcc-006600af37ff\") "
Oct 30 19:46:42.409907 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409691    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-tmp-dir\") pod \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\" (UID: \"d1a3b2eb-8722-4782-af07-70239b3c0ff5\") "
Oct 30 19:46:42.409907 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.409881    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-home" pod "d1a3b2eb-8722-4782-af07-70239b3c0ff5" (UID: "d1a3b2eb-8722-4782-af07-70239b3c0ff5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:42.410294 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.410101    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-tmp-dir" pod "d1a3b2eb-8722-4782-af07-70239b3c0ff5" (UID: "d1a3b2eb-8722-4782-af07-70239b3c0ff5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:42.410483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.410458    4712 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-tmp-dir\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:42.410559 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.410491    4712 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-model-cache\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:42.410559 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.410504    4712 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-home\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:42.410559 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.410518    4712 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-model-cache\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:42.410559 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.410531    4712 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-model-cache\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:42.411674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.411610    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-dshm" pod "aa037781-1c62-492d-adcc-006600af37ff" (UID: "aa037781-1c62-492d-adcc-006600af37ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:42.411674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.411658    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-home" pod "cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" (UID: "cdc58f78-58ef-4981-b1a9-4c6aa8b781fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:42.411863 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.411835    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa037781-1c62-492d-adcc-006600af37ff-kube-api-access-kknxw" pod "aa037781-1c62-492d-adcc-006600af37ff" (UID: "aa037781-1c62-492d-adcc-006600af37ff"). InnerVolumeSpecName "kube-api-access-kknxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:42.411928 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.411867    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-home" pod "aa037781-1c62-492d-adcc-006600af37ff" (UID: "aa037781-1c62-492d-adcc-006600af37ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:42.411928 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.411906    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-tmp-dir" pod "aa037781-1c62-492d-adcc-006600af37ff" (UID: "aa037781-1c62-492d-adcc-006600af37ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:42.412075 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.412052    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-tmp-dir" pod "cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" (UID: "cdc58f78-58ef-4981-b1a9-4c6aa8b781fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:42.412399 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.412333    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-dshm" pod "d1a3b2eb-8722-4782-af07-70239b3c0ff5" (UID: "d1a3b2eb-8722-4782-af07-70239b3c0ff5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:42.412500 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.412421    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-tls-certs" pod "cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" (UID: "cdc58f78-58ef-4981-b1a9-4c6aa8b781fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:42.412569 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.412536    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a3b2eb-8722-4782-af07-70239b3c0ff5-kube-api-access-l499v" pod "d1a3b2eb-8722-4782-af07-70239b3c0ff5" (UID: "d1a3b2eb-8722-4782-af07-70239b3c0ff5"). InnerVolumeSpecName "kube-api-access-l499v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:42.412892 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.412869    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-dshm" pod "cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" (UID: "cdc58f78-58ef-4981-b1a9-4c6aa8b781fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:42.413165 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.413148    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-kube-api-access-jdqm8" pod "cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" (UID: "cdc58f78-58ef-4981-b1a9-4c6aa8b781fc"). InnerVolumeSpecName "kube-api-access-jdqm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:42.413792 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.413767    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a3b2eb-8722-4782-af07-70239b3c0ff5-tls-certs" pod "d1a3b2eb-8722-4782-af07-70239b3c0ff5" (UID: "d1a3b2eb-8722-4782-af07-70239b3c0ff5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:42.414127 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.414107    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa037781-1c62-492d-adcc-006600af37ff-tls-certs" pod "aa037781-1c62-492d-adcc-006600af37ff" (UID: "aa037781-1c62-492d-adcc-006600af37ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:42.512316 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.512282    4712 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a3b2eb-8722-4782-af07-70239b3c0ff5-tls-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:42.512316 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.512312    4712 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a3b2eb-8722-4782-af07-70239b3c0ff5-dshm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:42.512316 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.512321    4712 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-tmp-dir\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:42.512560 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.512331    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdqm8\" (UniqueName: \"kubernetes.io/projected/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-kube-api-access-jdqm8\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:42.512560 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.512347    4712 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-home\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:42.512560 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.512356    4712 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-home\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:42.512560 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.512389    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l499v\" (UniqueName: \"kubernetes.io/projected/d1a3b2eb-8722-4782-af07-70239b3c0ff5-kube-api-access-l499v\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:42.512560 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.512401    4712 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-dshm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:42.512560 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.512413    4712 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-tls-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:42.512560 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.512424    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kknxw\" (UniqueName: \"kubernetes.io/projected/aa037781-1c62-492d-adcc-006600af37ff-kube-api-access-kknxw\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:42.512560 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.512434    4712 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc-dshm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:42.512560 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.512446    4712 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa037781-1c62-492d-adcc-006600af37ff-tmp-dir\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:42.512560 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.512457    4712 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa037781-1c62-492d-adcc-006600af37ff-tls-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:42.571468 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.571431    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:42.881388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.870873    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3cd29f0-6462-40b1-a5a0-30deec5e9321" path="/var/lib/kubelet/pods/b3cd29f0-6462-40b1-a5a0-30deec5e9321/volumes"
Oct 30 19:46:42.962177 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.962142    4712 generic.go:356] "Generic (PLEG): container finished" podID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerID="780f333b11a7f7aec3ab0b2d320191cf3a68df4a8a6b408cc3cc19110fa4ed04" exitCode=0
Oct 30 19:46:42.962371 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.962188    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"
Oct 30 19:46:42.962371 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.962221    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" event={"ID":"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc","Type":"ContainerDied","Data":"780f333b11a7f7aec3ab0b2d320191cf3a68df4a8a6b408cc3cc19110fa4ed04"}
Oct 30 19:46:42.962371 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.962270    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq" event={"ID":"cdc58f78-58ef-4981-b1a9-4c6aa8b781fc","Type":"ContainerDied","Data":"cf569081b22d7627213ba9d7437a45775f130c1e05f407bdbdc87254880751be"}
Oct 30 19:46:42.962371 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.962287    4712 scope.go:122] "RemoveContainer" containerID="780f333b11a7f7aec3ab0b2d320191cf3a68df4a8a6b408cc3cc19110fa4ed04"
Oct 30 19:46:42.963857 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.963832    4712 generic.go:356] "Generic (PLEG): container finished" podID="aa037781-1c62-492d-adcc-006600af37ff" containerID="463c5b2a01fe2bb2d5910e2addaf577df50697445bbbdeabe50ed4b2dac3a2d6" exitCode=0
Oct 30 19:46:42.963977 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.963879    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"
Oct 30 19:46:42.963977 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.963912    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" event={"ID":"aa037781-1c62-492d-adcc-006600af37ff","Type":"ContainerDied","Data":"463c5b2a01fe2bb2d5910e2addaf577df50697445bbbdeabe50ed4b2dac3a2d6"}
Oct 30 19:46:42.963977 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.963951    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8" event={"ID":"aa037781-1c62-492d-adcc-006600af37ff","Type":"ContainerDied","Data":"5dee4b07702292c330a748c509dfe0bb09d832d820c3fad9c9b058d73356d19f"}
Oct 30 19:46:42.965573 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.965546    4712 generic.go:356] "Generic (PLEG): container finished" podID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerID="b715774721260001c1debf173fc547a728f359b9dc8ac5b2465eb7ffeb926069" exitCode=0
Oct 30 19:46:42.965696 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.965600    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"
Oct 30 19:46:42.965696 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.965629    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" event={"ID":"d1a3b2eb-8722-4782-af07-70239b3c0ff5","Type":"ContainerDied","Data":"b715774721260001c1debf173fc547a728f359b9dc8ac5b2465eb7ffeb926069"}
Oct 30 19:46:42.965696 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.965674    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk" event={"ID":"d1a3b2eb-8722-4782-af07-70239b3c0ff5","Type":"ContainerDied","Data":"33b4e263125708550639faf29bc78d4d63860459b5f69f4388906bf51db91293"}
Oct 30 19:46:42.973337 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.973170    4712 scope.go:122] "RemoveContainer" containerID="780f333b11a7f7aec3ab0b2d320191cf3a68df4a8a6b408cc3cc19110fa4ed04"
Oct 30 19:46:42.973619 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:46:42.973450    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"780f333b11a7f7aec3ab0b2d320191cf3a68df4a8a6b408cc3cc19110fa4ed04\": container with ID starting with 780f333b11a7f7aec3ab0b2d320191cf3a68df4a8a6b408cc3cc19110fa4ed04 not found: ID does not exist" containerID="780f333b11a7f7aec3ab0b2d320191cf3a68df4a8a6b408cc3cc19110fa4ed04"
Oct 30 19:46:42.973619 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.973474    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780f333b11a7f7aec3ab0b2d320191cf3a68df4a8a6b408cc3cc19110fa4ed04"} err="failed to get container status \"780f333b11a7f7aec3ab0b2d320191cf3a68df4a8a6b408cc3cc19110fa4ed04\": rpc error: code = NotFound desc = could not find container \"780f333b11a7f7aec3ab0b2d320191cf3a68df4a8a6b408cc3cc19110fa4ed04\": container with ID starting with 780f333b11a7f7aec3ab0b2d320191cf3a68df4a8a6b408cc3cc19110fa4ed04 not found: ID does not exist"
Oct 30 19:46:42.973619 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.973493    4712 scope.go:122] "RemoveContainer" containerID="463c5b2a01fe2bb2d5910e2addaf577df50697445bbbdeabe50ed4b2dac3a2d6"
Oct 30 19:46:42.981602 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.981586    4712 scope.go:122] "RemoveContainer" containerID="463c5b2a01fe2bb2d5910e2addaf577df50697445bbbdeabe50ed4b2dac3a2d6"
Oct 30 19:46:42.981867 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:46:42.981849    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"463c5b2a01fe2bb2d5910e2addaf577df50697445bbbdeabe50ed4b2dac3a2d6\": container with ID starting with 463c5b2a01fe2bb2d5910e2addaf577df50697445bbbdeabe50ed4b2dac3a2d6 not found: ID does not exist" containerID="463c5b2a01fe2bb2d5910e2addaf577df50697445bbbdeabe50ed4b2dac3a2d6"
Oct 30 19:46:42.981923 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.981874    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"463c5b2a01fe2bb2d5910e2addaf577df50697445bbbdeabe50ed4b2dac3a2d6"} err="failed to get container status \"463c5b2a01fe2bb2d5910e2addaf577df50697445bbbdeabe50ed4b2dac3a2d6\": rpc error: code = NotFound desc = could not find container \"463c5b2a01fe2bb2d5910e2addaf577df50697445bbbdeabe50ed4b2dac3a2d6\": container with ID starting with 463c5b2a01fe2bb2d5910e2addaf577df50697445bbbdeabe50ed4b2dac3a2d6 not found: ID does not exist"
Oct 30 19:46:42.981923 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.981890    4712 scope.go:122] "RemoveContainer" containerID="b715774721260001c1debf173fc547a728f359b9dc8ac5b2465eb7ffeb926069"
Oct 30 19:46:42.990343 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.990326    4712 scope.go:122] "RemoveContainer" containerID="b715774721260001c1debf173fc547a728f359b9dc8ac5b2465eb7ffeb926069"
Oct 30 19:46:42.990629 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:46:42.990611    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b715774721260001c1debf173fc547a728f359b9dc8ac5b2465eb7ffeb926069\": container with ID starting with b715774721260001c1debf173fc547a728f359b9dc8ac5b2465eb7ffeb926069 not found: ID does not exist" containerID="b715774721260001c1debf173fc547a728f359b9dc8ac5b2465eb7ffeb926069"
Oct 30 19:46:42.990691 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:42.990638    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b715774721260001c1debf173fc547a728f359b9dc8ac5b2465eb7ffeb926069"} err="failed to get container status \"b715774721260001c1debf173fc547a728f359b9dc8ac5b2465eb7ffeb926069\": rpc error: code = NotFound desc = could not find container \"b715774721260001c1debf173fc547a728f359b9dc8ac5b2465eb7ffeb926069\": container with ID starting with b715774721260001c1debf173fc547a728f359b9dc8ac5b2465eb7ffeb926069 not found: ID does not exist"
Oct 30 19:46:43.020719 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:43.020699    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"]
Oct 30 19:46:43.032438 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:43.032418    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4b4c34/simulator-w0-4b4c34-kserve-f5fcbb459-zcclk"]
Oct 30 19:46:43.056002 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:43.055979    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"]
Oct 30 19:46:43.062062 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:43.062045    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4b4c34/distinct-w0-4b4c34-kserve-66c49c5849-55dlq"]
Oct 30 19:46:43.098254 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:43.098235    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"]
Oct 30 19:46:43.105262 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:43.105244    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4b4c34/distinct-2-w0-4b4c34-kserve-6c7bc7dc8-h7dq8"]
Oct 30 19:46:44.029623 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:44.029592    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:44.852787 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:44.852749    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa037781-1c62-492d-adcc-006600af37ff" path="/var/lib/kubelet/pods/aa037781-1c62-492d-adcc-006600af37ff/volumes"
Oct 30 19:46:44.853224 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:44.853208    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" path="/var/lib/kubelet/pods/cdc58f78-58ef-4981-b1a9-4c6aa8b781fc/volumes"
Oct 30 19:46:44.853674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:44.853659    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" path="/var/lib/kubelet/pods/d1a3b2eb-8722-4782-af07-70239b3c0ff5/volumes"
Oct 30 19:46:45.685552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:45.685519    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:46.204823 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:46.204796    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:46.468726 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:46.468624    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:46.657097 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:46.657066    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:47.752468 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:47.752376    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:47.881240 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:47.881213    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4"]
Oct 30 19:46:48.077091 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:48.076962    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:48.266522 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:48.266487    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:48.978245 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:48.978168    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:49.150526 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:49.150491    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:49.244760 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:49.244680    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"]
Oct 30 19:46:49.244939 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:49.244917    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5" podUID="b8c1988d-3a60-4e33-b9f2-c9878beb006b" containerName="istio-proxy" containerID="cri-o://43c6b1e76bc887c479abb46e7f5409e163543947cf7f170e262164ae676ada3d" gracePeriod=30
Oct 30 19:46:49.253805 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:49.253781    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"]
Oct 30 19:46:49.253989 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:49.253968    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7" podUID="d2fa6bc2-624f-4017-898f-2317c9064c73" containerName="istio-proxy" containerID="cri-o://205793fcae347b200b1755d5f49f4eeafaefc3aeedb68e7b2d547b0381b46886" gracePeriod=30
Oct 30 19:46:49.257483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:49.257462    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"]
Oct 30 19:46:49.257705 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:49.257677    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc" podUID="b727df7a-56c7-46fc-974a-38aebc2593c3" containerName="istio-proxy" containerID="cri-o://b13b462d93ac4b7cea8d62b6520164d674c301d00fc903e9b6e063ed7b4e5d03" gracePeriod=30
Oct 30 19:46:49.315908 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:49.315881    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:49.487281 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:49.487248    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:51.195051 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:51.195023    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:52.127485 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:52.127443    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:46:52.127665 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:52.127508    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7" podUID="d2fa6bc2-624f-4017-898f-2317c9064c73" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:46:52.530121 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:52.530040    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:52.882144 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:52.882107    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4" podUID="62d985a2-697f-4a97-a50b-fdecff1d5960" containerName="payload-pre-processing" containerID="cri-o://e3d1ab08c62a68baefc2dbf1189768c61fcdae2dc1a18ccb79a25dd84f8a99ac" gracePeriod=25
Oct 30 19:46:53.031206 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:53.031165    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:53.037683 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:53.037654    4712 generic.go:356] "Generic (PLEG): container finished" podID="62d985a2-697f-4a97-a50b-fdecff1d5960" containerID="e3d1ab08c62a68baefc2dbf1189768c61fcdae2dc1a18ccb79a25dd84f8a99ac" exitCode=0
Oct 30 19:46:53.037857 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:53.037724    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4" event={"ID":"62d985a2-697f-4a97-a50b-fdecff1d5960","Type":"ContainerDied","Data":"e3d1ab08c62a68baefc2dbf1189768c61fcdae2dc1a18ccb79a25dd84f8a99ac"}
Oct 30 19:46:53.151972 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:53.151949    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4"
Oct 30 19:46:53.193999 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:53.192885    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:53.218329 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:53.218278    4712 patch_prober.go:29] interesting pod/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:46:53.218510 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:53.218343    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5" podUID="b8c1988d-3a60-4e33-b9f2-c9878beb006b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:46:53.264963 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:53.264936    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d985a2-697f-4a97-a50b-fdecff1d5960-kube-api-access-jc4ct\" (UniqueName: \"kubernetes.io/projected/62d985a2-697f-4a97-a50b-fdecff1d5960-kube-api-access-jc4ct\") pod \"62d985a2-697f-4a97-a50b-fdecff1d5960\" (UID: \"62d985a2-697f-4a97-a50b-fdecff1d5960\") "
Oct 30 19:46:53.265132 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:53.265092    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62d985a2-697f-4a97-a50b-fdecff1d5960-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d985a2-697f-4a97-a50b-fdecff1d5960-plugins-config-volume\") pod \"62d985a2-697f-4a97-a50b-fdecff1d5960\" (UID: \"62d985a2-697f-4a97-a50b-fdecff1d5960\") "
Oct 30 19:46:53.265563 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:53.265534    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d985a2-697f-4a97-a50b-fdecff1d5960-plugins-config-volume" pod "62d985a2-697f-4a97-a50b-fdecff1d5960" (UID: "62d985a2-697f-4a97-a50b-fdecff1d5960"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:53.265663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:53.265641    4712 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d985a2-697f-4a97-a50b-fdecff1d5960-plugins-config-volume\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:53.267037 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:53.267014    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d985a2-697f-4a97-a50b-fdecff1d5960-kube-api-access-jc4ct" pod "62d985a2-697f-4a97-a50b-fdecff1d5960" (UID: "62d985a2-697f-4a97-a50b-fdecff1d5960"). InnerVolumeSpecName "kube-api-access-jc4ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:53.367342 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:53.367314    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc4ct\" (UniqueName: \"kubernetes.io/projected/62d985a2-697f-4a97-a50b-fdecff1d5960-kube-api-access-jc4ct\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:53.483107 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:53.483017    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:53.740432 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:53.740301    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:54.044018 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.043942    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4" event={"ID":"62d985a2-697f-4a97-a50b-fdecff1d5960","Type":"ContainerDied","Data":"0454df5df5f94aa2d2d8b26a75b2633304ea6c1878bb8fb58537f0ca43455cc0"}
Oct 30 19:46:54.044018 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.043973    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4"
Oct 30 19:46:54.044018 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.044001    4712 scope.go:122] "RemoveContainer" containerID="e3d1ab08c62a68baefc2dbf1189768c61fcdae2dc1a18ccb79a25dd84f8a99ac"
Oct 30 19:46:54.097946 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.097921    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4"]
Oct 30 19:46:54.109657 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.109636    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-fe4ec3-784dff5459-s7pv4"]
Oct 30 19:46:54.539191 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.539163    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:46:54.545583 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.545563    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:46:54.581825 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.581804    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b727df7a-56c7-46fc-974a-38aebc2593c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b727df7a-56c7-46fc-974a-38aebc2593c3-istiod-ca-cert\") pod \"b727df7a-56c7-46fc-974a-38aebc2593c3\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") "
Oct 30 19:46:54.581935 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.581841    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-credential-socket\") pod \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") "
Oct 30 19:46:54.581935 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.581871    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-data\") pod \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") "
Oct 30 19:46:54.581935 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.581913    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-workload-certs\") pod \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") "
Oct 30 19:46:54.582104 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.581943    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-envoy\") pod \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") "
Oct 30 19:46:54.582104 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.581992    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-workload-certs\") pod \"b727df7a-56c7-46fc-974a-38aebc2593c3\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") "
Oct 30 19:46:54.582104 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.582053    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-token\" (UniqueName: \"kubernetes.io/projected/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-token\") pod \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") "
Oct 30 19:46:54.582104 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.582085    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-credential-socket\") pod \"b727df7a-56c7-46fc-974a-38aebc2593c3\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") "
Oct 30 19:46:54.582318 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.582119    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-podinfo\") pod \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") "
Oct 30 19:46:54.582318 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.582137    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-data" pod "b8c1988d-3a60-4e33-b9f2-c9878beb006b" (UID: "b8c1988d-3a60-4e33-b9f2-c9878beb006b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:54.582318 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.582145    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-podinfo\") pod \"b727df7a-56c7-46fc-974a-38aebc2593c3\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") "
Oct 30 19:46:54.582318 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.582219    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-workload-socket\") pod \"b727df7a-56c7-46fc-974a-38aebc2593c3\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") "
Oct 30 19:46:54.582593 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.582371    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b727df7a-56c7-46fc-974a-38aebc2593c3-kube-api-access-llgnm\" (UniqueName: \"kubernetes.io/projected/b727df7a-56c7-46fc-974a-38aebc2593c3-kube-api-access-llgnm\") pod \"b727df7a-56c7-46fc-974a-38aebc2593c3\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") "
Oct 30 19:46:54.582593 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.582164    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-credential-socket" pod "b8c1988d-3a60-4e33-b9f2-c9878beb006b" (UID: "b8c1988d-3a60-4e33-b9f2-c9878beb006b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:54.582593 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.582193    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b727df7a-56c7-46fc-974a-38aebc2593c3-istiod-ca-cert" pod "b727df7a-56c7-46fc-974a-38aebc2593c3" (UID: "b727df7a-56c7-46fc-974a-38aebc2593c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:54.582593 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.582454    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-credential-socket" pod "b727df7a-56c7-46fc-974a-38aebc2593c3" (UID: "b727df7a-56c7-46fc-974a-38aebc2593c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:54.582593 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.582548    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-workload-certs" pod "b727df7a-56c7-46fc-974a-38aebc2593c3" (UID: "b727df7a-56c7-46fc-974a-38aebc2593c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:54.582593 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.582555    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-workload-socket\") pod \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") "
Oct 30 19:46:54.582906 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.582596    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-workload-certs" pod "b8c1988d-3a60-4e33-b9f2-c9878beb006b" (UID: "b8c1988d-3a60-4e33-b9f2-c9878beb006b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:54.582906 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.582669    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istiod-ca-cert\") pod \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") "
Oct 30 19:46:54.582906 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.582749    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-token\" (UniqueName: \"kubernetes.io/projected/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-token\") pod \"b727df7a-56c7-46fc-974a-38aebc2593c3\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") "
Oct 30 19:46:54.582906 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.582814    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-data\") pod \"b727df7a-56c7-46fc-974a-38aebc2593c3\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") "
Oct 30 19:46:54.582906 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.582873    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-envoy\") pod \"b727df7a-56c7-46fc-974a-38aebc2593c3\" (UID: \"b727df7a-56c7-46fc-974a-38aebc2593c3\") "
Oct 30 19:46:54.583158 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.582916    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c1988d-3a60-4e33-b9f2-c9878beb006b-kube-api-access-5scbd\" (UniqueName: \"kubernetes.io/projected/b8c1988d-3a60-4e33-b9f2-c9878beb006b-kube-api-access-5scbd\") pod \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\" (UID: \"b8c1988d-3a60-4e33-b9f2-c9878beb006b\") "
Oct 30 19:46:54.583451 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.583425    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-workload-socket" pod "b8c1988d-3a60-4e33-b9f2-c9878beb006b" (UID: "b8c1988d-3a60-4e33-b9f2-c9878beb006b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:54.584036 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.583684    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.584036 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.583712    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.584036 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.583729    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.584036 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.583745    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b727df7a-56c7-46fc-974a-38aebc2593c3-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.584036 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.583759    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.584036 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.583805    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.584036 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.583821    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.585264 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.584189    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-data" pod "b727df7a-56c7-46fc-974a-38aebc2593c3" (UID: "b727df7a-56c7-46fc-974a-38aebc2593c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:54.585264 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.584299    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istiod-ca-cert" pod "b8c1988d-3a60-4e33-b9f2-c9878beb006b" (UID: "b8c1988d-3a60-4e33-b9f2-c9878beb006b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:54.585264 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.584805    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-workload-socket" pod "b727df7a-56c7-46fc-974a-38aebc2593c3" (UID: "b727df7a-56c7-46fc-974a-38aebc2593c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:54.590096 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.590068    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-envoy" pod "b8c1988d-3a60-4e33-b9f2-c9878beb006b" (UID: "b8c1988d-3a60-4e33-b9f2-c9878beb006b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:54.590749 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.590184    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b727df7a-56c7-46fc-974a-38aebc2593c3-kube-api-access-llgnm" pod "b727df7a-56c7-46fc-974a-38aebc2593c3" (UID: "b727df7a-56c7-46fc-974a-38aebc2593c3"). InnerVolumeSpecName "kube-api-access-llgnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:54.590749 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.590420    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c1988d-3a60-4e33-b9f2-c9878beb006b-kube-api-access-5scbd" pod "b8c1988d-3a60-4e33-b9f2-c9878beb006b" (UID: "b8c1988d-3a60-4e33-b9f2-c9878beb006b"). InnerVolumeSpecName "kube-api-access-5scbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:54.590749 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.590512    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-podinfo" pod "b8c1988d-3a60-4e33-b9f2-c9878beb006b" (UID: "b8c1988d-3a60-4e33-b9f2-c9878beb006b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:54.590956 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.590827    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-token" pod "b8c1988d-3a60-4e33-b9f2-c9878beb006b" (UID: "b8c1988d-3a60-4e33-b9f2-c9878beb006b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:54.591766 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.591736    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-envoy" pod "b727df7a-56c7-46fc-974a-38aebc2593c3" (UID: "b727df7a-56c7-46fc-974a-38aebc2593c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:54.592757 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.592700    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-podinfo" pod "b727df7a-56c7-46fc-974a-38aebc2593c3" (UID: "b727df7a-56c7-46fc-974a-38aebc2593c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:54.592848 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.592779    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:46:54.598872 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.598851    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-token" pod "b727df7a-56c7-46fc-974a-38aebc2593c3" (UID: "b727df7a-56c7-46fc-974a-38aebc2593c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:54.684882 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.684848    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-workload-socket\") pod \"d2fa6bc2-624f-4017-898f-2317c9064c73\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") "
Oct 30 19:46:54.685057 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.684905    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-workload-certs\") pod \"d2fa6bc2-624f-4017-898f-2317c9064c73\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") "
Oct 30 19:46:54.685057 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.684952    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-podinfo\") pod \"d2fa6bc2-624f-4017-898f-2317c9064c73\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") "
Oct 30 19:46:54.685057 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.685008    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2fa6bc2-624f-4017-898f-2317c9064c73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2fa6bc2-624f-4017-898f-2317c9064c73-istiod-ca-cert\") pod \"d2fa6bc2-624f-4017-898f-2317c9064c73\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") "
Oct 30 19:46:54.685057 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.685042    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-credential-socket\") pod \"d2fa6bc2-624f-4017-898f-2317c9064c73\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") "
Oct 30 19:46:54.685268 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.685089    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2fa6bc2-624f-4017-898f-2317c9064c73-kube-api-access-kmgd9\" (UniqueName: \"kubernetes.io/projected/d2fa6bc2-624f-4017-898f-2317c9064c73-kube-api-access-kmgd9\") pod \"d2fa6bc2-624f-4017-898f-2317c9064c73\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") "
Oct 30 19:46:54.685268 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.685178    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-workload-socket" pod "d2fa6bc2-624f-4017-898f-2317c9064c73" (UID: "d2fa6bc2-624f-4017-898f-2317c9064c73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:54.685268 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.685215    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-data\") pod \"d2fa6bc2-624f-4017-898f-2317c9064c73\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") "
Oct 30 19:46:54.685268 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.685237    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-workload-certs" pod "d2fa6bc2-624f-4017-898f-2317c9064c73" (UID: "d2fa6bc2-624f-4017-898f-2317c9064c73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:54.685504 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.685281    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-token\" (UniqueName: \"kubernetes.io/projected/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-token\") pod \"d2fa6bc2-624f-4017-898f-2317c9064c73\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") "
Oct 30 19:46:54.685504 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.685322    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-envoy\") pod \"d2fa6bc2-624f-4017-898f-2317c9064c73\" (UID: \"d2fa6bc2-624f-4017-898f-2317c9064c73\") "
Oct 30 19:46:54.685504 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.685411    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-credential-socket" pod "d2fa6bc2-624f-4017-898f-2317c9064c73" (UID: "d2fa6bc2-624f-4017-898f-2317c9064c73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:54.685638 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.685529    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-data" pod "d2fa6bc2-624f-4017-898f-2317c9064c73" (UID: "d2fa6bc2-624f-4017-898f-2317c9064c73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:54.686447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.685959    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2fa6bc2-624f-4017-898f-2317c9064c73-istiod-ca-cert" pod "d2fa6bc2-624f-4017-898f-2317c9064c73" (UID: "d2fa6bc2-624f-4017-898f-2317c9064c73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:54.686447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.685967    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.686447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.686015    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.686447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.686029    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.686447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.686042    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.686447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.686055    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.686447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.686069    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.686447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.686085    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.686447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.686098    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.686447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.686112    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llgnm\" (UniqueName: \"kubernetes.io/projected/b727df7a-56c7-46fc-974a-38aebc2593c3-kube-api-access-llgnm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.686447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.686126    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.686447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.686139    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c1988d-3a60-4e33-b9f2-c9878beb006b-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.686447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.686152    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.686447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.686166    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.686447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.686179    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b727df7a-56c7-46fc-974a-38aebc2593c3-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.686447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.686193    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5scbd\" (UniqueName: \"kubernetes.io/projected/b8c1988d-3a60-4e33-b9f2-c9878beb006b-kube-api-access-5scbd\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.687869 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.687849    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-podinfo" pod "d2fa6bc2-624f-4017-898f-2317c9064c73" (UID: "d2fa6bc2-624f-4017-898f-2317c9064c73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:54.687948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.687905    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2fa6bc2-624f-4017-898f-2317c9064c73-kube-api-access-kmgd9" pod "d2fa6bc2-624f-4017-898f-2317c9064c73" (UID: "d2fa6bc2-624f-4017-898f-2317c9064c73"). InnerVolumeSpecName "kube-api-access-kmgd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:54.688003 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.687986    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-token" pod "d2fa6bc2-624f-4017-898f-2317c9064c73" (UID: "d2fa6bc2-624f-4017-898f-2317c9064c73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:54.688078 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.688065    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-envoy" pod "d2fa6bc2-624f-4017-898f-2317c9064c73" (UID: "d2fa6bc2-624f-4017-898f-2317c9064c73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:54.786993 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.786967    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.786993 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.786990    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2fa6bc2-624f-4017-898f-2317c9064c73-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.786993 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.787000    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmgd9\" (UniqueName: \"kubernetes.io/projected/d2fa6bc2-624f-4017-898f-2317c9064c73-kube-api-access-kmgd9\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.787420 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.787009    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.787420 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.787018    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2fa6bc2-624f-4017-898f-2317c9064c73-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:46:54.853056 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:54.852991    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62d985a2-697f-4a97-a50b-fdecff1d5960" path="/var/lib/kubelet/pods/62d985a2-697f-4a97-a50b-fdecff1d5960/volumes"
Oct 30 19:46:55.003957 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.003921    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:55.052730 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.052694    4712 generic.go:356] "Generic (PLEG): container finished" podID="d2fa6bc2-624f-4017-898f-2317c9064c73" containerID="205793fcae347b200b1755d5f49f4eeafaefc3aeedb68e7b2d547b0381b46886" exitCode=0
Oct 30 19:46:55.052906 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.052737    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"
Oct 30 19:46:55.052906 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.052780    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7" event={"ID":"d2fa6bc2-624f-4017-898f-2317c9064c73","Type":"ContainerDied","Data":"205793fcae347b200b1755d5f49f4eeafaefc3aeedb68e7b2d547b0381b46886"}
Oct 30 19:46:55.052906 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.052854    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7" event={"ID":"d2fa6bc2-624f-4017-898f-2317c9064c73","Type":"ContainerDied","Data":"f231de1d36dd03f857fd19b3f29c2406119179a9fc0afdb62ba712e1b8f5f86d"}
Oct 30 19:46:55.052906 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.052888    4712 scope.go:122] "RemoveContainer" containerID="205793fcae347b200b1755d5f49f4eeafaefc3aeedb68e7b2d547b0381b46886"
Oct 30 19:46:55.054909 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.054878    4712 generic.go:356] "Generic (PLEG): container finished" podID="b727df7a-56c7-46fc-974a-38aebc2593c3" containerID="b13b462d93ac4b7cea8d62b6520164d674c301d00fc903e9b6e063ed7b4e5d03" exitCode=0
Oct 30 19:46:55.055022 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.054983    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc" event={"ID":"b727df7a-56c7-46fc-974a-38aebc2593c3","Type":"ContainerDied","Data":"b13b462d93ac4b7cea8d62b6520164d674c301d00fc903e9b6e063ed7b4e5d03"}
Oct 30 19:46:55.055097 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.055047    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc" event={"ID":"b727df7a-56c7-46fc-974a-38aebc2593c3","Type":"ContainerDied","Data":"8fe4b2a018f9153a287115040c14625f1d5ef931bde8dd12ff0ca07083dcaad0"}
Oct 30 19:46:55.055255 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.055238    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"
Oct 30 19:46:55.058234 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.058210    4712 generic.go:356] "Generic (PLEG): container finished" podID="b8c1988d-3a60-4e33-b9f2-c9878beb006b" containerID="43c6b1e76bc887c479abb46e7f5409e163543947cf7f170e262164ae676ada3d" exitCode=0
Oct 30 19:46:55.058316 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.058250    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"
Oct 30 19:46:55.058316 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.058287    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5" event={"ID":"b8c1988d-3a60-4e33-b9f2-c9878beb006b","Type":"ContainerDied","Data":"43c6b1e76bc887c479abb46e7f5409e163543947cf7f170e262164ae676ada3d"}
Oct 30 19:46:55.058441 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.058329    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5" event={"ID":"b8c1988d-3a60-4e33-b9f2-c9878beb006b","Type":"ContainerDied","Data":"9302bb8386063a07ada4392a720274e3677537e361e31f7798e2e3854c262be2"}
Oct 30 19:46:55.063519 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.063490    4712 scope.go:122] "RemoveContainer" containerID="205793fcae347b200b1755d5f49f4eeafaefc3aeedb68e7b2d547b0381b46886"
Oct 30 19:46:55.063805 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:46:55.063778    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"205793fcae347b200b1755d5f49f4eeafaefc3aeedb68e7b2d547b0381b46886\": container with ID starting with 205793fcae347b200b1755d5f49f4eeafaefc3aeedb68e7b2d547b0381b46886 not found: ID does not exist" containerID="205793fcae347b200b1755d5f49f4eeafaefc3aeedb68e7b2d547b0381b46886"
Oct 30 19:46:55.063907 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.063813    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"205793fcae347b200b1755d5f49f4eeafaefc3aeedb68e7b2d547b0381b46886"} err="failed to get container status \"205793fcae347b200b1755d5f49f4eeafaefc3aeedb68e7b2d547b0381b46886\": rpc error: code = NotFound desc = could not find container \"205793fcae347b200b1755d5f49f4eeafaefc3aeedb68e7b2d547b0381b46886\": container with ID starting with 205793fcae347b200b1755d5f49f4eeafaefc3aeedb68e7b2d547b0381b46886 not found: ID does not exist"
Oct 30 19:46:55.063907 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.063830    4712 scope.go:122] "RemoveContainer" containerID="b13b462d93ac4b7cea8d62b6520164d674c301d00fc903e9b6e063ed7b4e5d03"
Oct 30 19:46:55.072601 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.072581    4712 scope.go:122] "RemoveContainer" containerID="9b53887b9a54a31204a0e16b19ff4331e4dd29b74539b697a6bf066b43cfc056"
Oct 30 19:46:55.081010 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.080995    4712 scope.go:122] "RemoveContainer" containerID="b13b462d93ac4b7cea8d62b6520164d674c301d00fc903e9b6e063ed7b4e5d03"
Oct 30 19:46:55.081269 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:46:55.081249    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b13b462d93ac4b7cea8d62b6520164d674c301d00fc903e9b6e063ed7b4e5d03\": container with ID starting with b13b462d93ac4b7cea8d62b6520164d674c301d00fc903e9b6e063ed7b4e5d03 not found: ID does not exist" containerID="b13b462d93ac4b7cea8d62b6520164d674c301d00fc903e9b6e063ed7b4e5d03"
Oct 30 19:46:55.081350 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.081280    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b13b462d93ac4b7cea8d62b6520164d674c301d00fc903e9b6e063ed7b4e5d03"} err="failed to get container status \"b13b462d93ac4b7cea8d62b6520164d674c301d00fc903e9b6e063ed7b4e5d03\": rpc error: code = NotFound desc = could not find container \"b13b462d93ac4b7cea8d62b6520164d674c301d00fc903e9b6e063ed7b4e5d03\": container with ID starting with b13b462d93ac4b7cea8d62b6520164d674c301d00fc903e9b6e063ed7b4e5d03 not found: ID does not exist"
Oct 30 19:46:55.081350 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.081301    4712 scope.go:122] "RemoveContainer" containerID="9b53887b9a54a31204a0e16b19ff4331e4dd29b74539b697a6bf066b43cfc056"
Oct 30 19:46:55.081691 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:46:55.081672    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b53887b9a54a31204a0e16b19ff4331e4dd29b74539b697a6bf066b43cfc056\": container with ID starting with 9b53887b9a54a31204a0e16b19ff4331e4dd29b74539b697a6bf066b43cfc056 not found: ID does not exist" containerID="9b53887b9a54a31204a0e16b19ff4331e4dd29b74539b697a6bf066b43cfc056"
Oct 30 19:46:55.081744 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.081697    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b53887b9a54a31204a0e16b19ff4331e4dd29b74539b697a6bf066b43cfc056"} err="failed to get container status \"9b53887b9a54a31204a0e16b19ff4331e4dd29b74539b697a6bf066b43cfc056\": rpc error: code = NotFound desc = could not find container \"9b53887b9a54a31204a0e16b19ff4331e4dd29b74539b697a6bf066b43cfc056\": container with ID starting with 9b53887b9a54a31204a0e16b19ff4331e4dd29b74539b697a6bf066b43cfc056 not found: ID does not exist"
Oct 30 19:46:55.081744 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.081711    4712 scope.go:122] "RemoveContainer" containerID="43c6b1e76bc887c479abb46e7f5409e163543947cf7f170e262164ae676ada3d"
Oct 30 19:46:55.093909 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.093893    4712 scope.go:122] "RemoveContainer" containerID="43c6b1e76bc887c479abb46e7f5409e163543947cf7f170e262164ae676ada3d"
Oct 30 19:46:55.094154 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:46:55.094139    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43c6b1e76bc887c479abb46e7f5409e163543947cf7f170e262164ae676ada3d\": container with ID starting with 43c6b1e76bc887c479abb46e7f5409e163543947cf7f170e262164ae676ada3d not found: ID does not exist" containerID="43c6b1e76bc887c479abb46e7f5409e163543947cf7f170e262164ae676ada3d"
Oct 30 19:46:55.094203 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.094159    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43c6b1e76bc887c479abb46e7f5409e163543947cf7f170e262164ae676ada3d"} err="failed to get container status \"43c6b1e76bc887c479abb46e7f5409e163543947cf7f170e262164ae676ada3d\": rpc error: code = NotFound desc = could not find container \"43c6b1e76bc887c479abb46e7f5409e163543947cf7f170e262164ae676ada3d\": container with ID starting with 43c6b1e76bc887c479abb46e7f5409e163543947cf7f170e262164ae676ada3d not found: ID does not exist"
Oct 30 19:46:55.109087 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.109039    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"]
Oct 30 19:46:55.116494 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.116473    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-9m9j5"]
Oct 30 19:46:55.149606 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.149587    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"]
Oct 30 19:46:55.156181 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.156162    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-s7qcc"]
Oct 30 19:46:55.196948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.196921    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"]
Oct 30 19:46:55.204830 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:55.204808    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef9a5a-openshift-default-7bd7f4b75f-4xmm7"]
Oct 30 19:46:56.480237 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:56.480205    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:56.851865 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:56.851786    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b727df7a-56c7-46fc-974a-38aebc2593c3" path="/var/lib/kubelet/pods/b727df7a-56c7-46fc-974a-38aebc2593c3/volumes"
Oct 30 19:46:56.852264 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:56.852250    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c1988d-3a60-4e33-b9f2-c9878beb006b" path="/var/lib/kubelet/pods/b8c1988d-3a60-4e33-b9f2-c9878beb006b/volumes"
Oct 30 19:46:56.852679 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:56.852666    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2fa6bc2-624f-4017-898f-2317c9064c73" path="/var/lib/kubelet/pods/d2fa6bc2-624f-4017-898f-2317c9064c73/volumes"
Oct 30 19:46:58.078563 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:58.078482    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:59.678270 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:59.678191    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:46:59.893960 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:46:59.893902    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:00.057691 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:00.057611    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:00.204435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:00.204398    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"]
Oct 30 19:47:00.204672 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:00.204642    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728" podUID="35cbcb48-e970-48e6-a942-05cd29730d9a" containerName="istio-proxy" containerID="cri-o://4c1ca5dbbd04f2b2d9073d93c768e2ad31953c04f28d53a82d26bd03cfe86c0b" gracePeriod=30
Oct 30 19:47:00.280339 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:00.280311    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:01.418720 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:01.418681    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:03.183415 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:03.183378    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:03.740402 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:03.740356    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:03.940513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:03.940479    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:04.150225 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:04.146913    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:04.412740 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:04.412666    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:05.472338 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.472280    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:47:05.620012 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.619984    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-token\" (UniqueName: \"kubernetes.io/projected/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-token\") pod \"35cbcb48-e970-48e6-a942-05cd29730d9a\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") "
Oct 30 19:47:05.620175 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.620021    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-podinfo\") pod \"35cbcb48-e970-48e6-a942-05cd29730d9a\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") "
Oct 30 19:47:05.620175 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.620114    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-envoy\") pod \"35cbcb48-e970-48e6-a942-05cd29730d9a\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") "
Oct 30 19:47:05.620175 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.620145    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35cbcb48-e970-48e6-a942-05cd29730d9a-kube-api-access-gsr85\" (UniqueName: \"kubernetes.io/projected/35cbcb48-e970-48e6-a942-05cd29730d9a-kube-api-access-gsr85\") pod \"35cbcb48-e970-48e6-a942-05cd29730d9a\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") "
Oct 30 19:47:05.620175 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.620169    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-data\") pod \"35cbcb48-e970-48e6-a942-05cd29730d9a\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") "
Oct 30 19:47:05.620398 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.620191    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35cbcb48-e970-48e6-a942-05cd29730d9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cbcb48-e970-48e6-a942-05cd29730d9a-istiod-ca-cert\") pod \"35cbcb48-e970-48e6-a942-05cd29730d9a\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") "
Oct 30 19:47:05.620398 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.620289    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-workload-socket\") pod \"35cbcb48-e970-48e6-a942-05cd29730d9a\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") "
Oct 30 19:47:05.620398 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.620329    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-credential-socket\") pod \"35cbcb48-e970-48e6-a942-05cd29730d9a\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") "
Oct 30 19:47:05.620398 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.620392    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-workload-certs\") pod \"35cbcb48-e970-48e6-a942-05cd29730d9a\" (UID: \"35cbcb48-e970-48e6-a942-05cd29730d9a\") "
Oct 30 19:47:05.620637 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.620514    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-data" pod "35cbcb48-e970-48e6-a942-05cd29730d9a" (UID: "35cbcb48-e970-48e6-a942-05cd29730d9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:05.620884 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.620795    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-workload-certs" pod "35cbcb48-e970-48e6-a942-05cd29730d9a" (UID: "35cbcb48-e970-48e6-a942-05cd29730d9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:05.620884 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.620820    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35cbcb48-e970-48e6-a942-05cd29730d9a-istiod-ca-cert" pod "35cbcb48-e970-48e6-a942-05cd29730d9a" (UID: "35cbcb48-e970-48e6-a942-05cd29730d9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:05.621106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.620979    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:05.621106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.621002    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:05.621106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.621015    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cbcb48-e970-48e6-a942-05cd29730d9a-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:05.621106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.621011    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-credential-socket" pod "35cbcb48-e970-48e6-a942-05cd29730d9a" (UID: "35cbcb48-e970-48e6-a942-05cd29730d9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:05.621564 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.621533    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-workload-socket" pod "35cbcb48-e970-48e6-a942-05cd29730d9a" (UID: "35cbcb48-e970-48e6-a942-05cd29730d9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:05.622325 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.622294    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-token" pod "35cbcb48-e970-48e6-a942-05cd29730d9a" (UID: "35cbcb48-e970-48e6-a942-05cd29730d9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:05.622325 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.622302    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-podinfo" pod "35cbcb48-e970-48e6-a942-05cd29730d9a" (UID: "35cbcb48-e970-48e6-a942-05cd29730d9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:05.622490 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.622324    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-envoy" pod "35cbcb48-e970-48e6-a942-05cd29730d9a" (UID: "35cbcb48-e970-48e6-a942-05cd29730d9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:05.622529 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.622488    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cbcb48-e970-48e6-a942-05cd29730d9a-kube-api-access-gsr85" pod "35cbcb48-e970-48e6-a942-05cd29730d9a" (UID: "35cbcb48-e970-48e6-a942-05cd29730d9a"). InnerVolumeSpecName "kube-api-access-gsr85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:05.638550 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.638527    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:05.722728 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.722664    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:05.722728 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.722688    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:05.722728 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.722698    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:05.722728 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.722708    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsr85\" (UniqueName: \"kubernetes.io/projected/35cbcb48-e970-48e6-a942-05cd29730d9a-kube-api-access-gsr85\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:05.722728 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.722717    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:05.722728 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:05.722727    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cbcb48-e970-48e6-a942-05cd29730d9a-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:06.128550 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:06.128519    4712 generic.go:356] "Generic (PLEG): container finished" podID="35cbcb48-e970-48e6-a942-05cd29730d9a" containerID="4c1ca5dbbd04f2b2d9073d93c768e2ad31953c04f28d53a82d26bd03cfe86c0b" exitCode=0
Oct 30 19:47:06.128716 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:06.128568    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"
Oct 30 19:47:06.128716 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:06.128579    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728" event={"ID":"35cbcb48-e970-48e6-a942-05cd29730d9a","Type":"ContainerDied","Data":"4c1ca5dbbd04f2b2d9073d93c768e2ad31953c04f28d53a82d26bd03cfe86c0b"}
Oct 30 19:47:06.128716 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:06.128622    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728" event={"ID":"35cbcb48-e970-48e6-a942-05cd29730d9a","Type":"ContainerDied","Data":"7cfca795ae738866e46f1860e91e541157798955e55ca8e60da29f291424a874"}
Oct 30 19:47:06.128716 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:06.128642    4712 scope.go:122] "RemoveContainer" containerID="4c1ca5dbbd04f2b2d9073d93c768e2ad31953c04f28d53a82d26bd03cfe86c0b"
Oct 30 19:47:06.138268 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:06.138252    4712 scope.go:122] "RemoveContainer" containerID="43bca3a96c5a5a944f8f368ca37aa14413f576d5ae25a935c72e00d1de452ac6"
Oct 30 19:47:06.147040 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:06.147022    4712 scope.go:122] "RemoveContainer" containerID="4c1ca5dbbd04f2b2d9073d93c768e2ad31953c04f28d53a82d26bd03cfe86c0b"
Oct 30 19:47:06.147297 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:47:06.147278    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c1ca5dbbd04f2b2d9073d93c768e2ad31953c04f28d53a82d26bd03cfe86c0b\": container with ID starting with 4c1ca5dbbd04f2b2d9073d93c768e2ad31953c04f28d53a82d26bd03cfe86c0b not found: ID does not exist" containerID="4c1ca5dbbd04f2b2d9073d93c768e2ad31953c04f28d53a82d26bd03cfe86c0b"
Oct 30 19:47:06.147391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:06.147312    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c1ca5dbbd04f2b2d9073d93c768e2ad31953c04f28d53a82d26bd03cfe86c0b"} err="failed to get container status \"4c1ca5dbbd04f2b2d9073d93c768e2ad31953c04f28d53a82d26bd03cfe86c0b\": rpc error: code = NotFound desc = could not find container \"4c1ca5dbbd04f2b2d9073d93c768e2ad31953c04f28d53a82d26bd03cfe86c0b\": container with ID starting with 4c1ca5dbbd04f2b2d9073d93c768e2ad31953c04f28d53a82d26bd03cfe86c0b not found: ID does not exist"
Oct 30 19:47:06.147391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:06.147336    4712 scope.go:122] "RemoveContainer" containerID="43bca3a96c5a5a944f8f368ca37aa14413f576d5ae25a935c72e00d1de452ac6"
Oct 30 19:47:06.147596 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:47:06.147581    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43bca3a96c5a5a944f8f368ca37aa14413f576d5ae25a935c72e00d1de452ac6\": container with ID starting with 43bca3a96c5a5a944f8f368ca37aa14413f576d5ae25a935c72e00d1de452ac6 not found: ID does not exist" containerID="43bca3a96c5a5a944f8f368ca37aa14413f576d5ae25a935c72e00d1de452ac6"
Oct 30 19:47:06.147636 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:06.147602    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43bca3a96c5a5a944f8f368ca37aa14413f576d5ae25a935c72e00d1de452ac6"} err="failed to get container status \"43bca3a96c5a5a944f8f368ca37aa14413f576d5ae25a935c72e00d1de452ac6\": rpc error: code = NotFound desc = could not find container \"43bca3a96c5a5a944f8f368ca37aa14413f576d5ae25a935c72e00d1de452ac6\": container with ID starting with 43bca3a96c5a5a944f8f368ca37aa14413f576d5ae25a935c72e00d1de452ac6 not found: ID does not exist"
Oct 30 19:47:06.215421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:06.215383    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"]
Oct 30 19:47:06.229465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:06.229437    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fe4ec3-openshift-default-5cc98576f7-qd728"]
Oct 30 19:47:06.373640 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:47:06.373607    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:47:06.630981 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:06.630945    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:06.852226 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:06.852194    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35cbcb48-e970-48e6-a942-05cd29730d9a" path="/var/lib/kubelet/pods/35cbcb48-e970-48e6-a942-05cd29730d9a/volumes"
Oct 30 19:47:06.972220 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:06.972147    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:07.368525 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:07.368501    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:07.643552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:07.643473    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:07.980293 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:07.980217    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:08.188760 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:08.188728    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:09.130076 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:09.130042    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:09.292028 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:09.291984    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:10.114214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:10.114178    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:10.383498 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:10.383467    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:10.916051 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:10.916018    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:11.675559 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:11.675524    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:12.041431 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:12.041330    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:12.735637 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:12.735608    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:13.476398 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:13.476350    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:13.661130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:13.661092    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:13.924379 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:13.924324    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:14.358946 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:14.358835    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:14.534985 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:14.534938    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:14.713330 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:14.713295    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:14.889540 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:14.889325    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"]
Oct 30 19:47:14.889749 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:14.889570    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst" podUID="045af806-47d7-41b6-9cd7-ab0b9c851f3c" containerName="istio-proxy" containerID="cri-o://e469737daf6cc305c44bf5292b0d9161b906490879a98081f4df75c7ad4067fa" gracePeriod=30
Oct 30 19:47:14.901537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:14.901507    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"]
Oct 30 19:47:14.901752 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:14.901729    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25" podUID="e42358a7-0f5b-476a-831a-cefabde3984e" containerName="istio-proxy" containerID="cri-o://5bb9d130295a6d0be7045bd5a9f972f411b9201c510849a6a868f634152476c2" gracePeriod=30
Oct 30 19:47:14.961514 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:14.961479    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:16.491310 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:16.491278    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:17.998459 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:17.998423    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"]
Oct 30 19:47:17.998875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:17.998621    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp" podUID="3fce9cac-38e0-436f-a4aa-4f82b908e11e" containerName="istio-proxy" containerID="cri-o://1fd0a1b7ca423ef3634ec9d8ae6a6f7ee03cdaf34f63a430faee2f5bc16da024" gracePeriod=30
Oct 30 19:47:18.221426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:18.221382    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:18.747805 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:18.747776    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:19.248421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:19.248389    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"]
Oct 30 19:47:19.411697 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:19.411666    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:19.787945 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:19.787911    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:20.162039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.162015    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:47:20.196031 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.196011    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:47:20.198244 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.198227    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-workload-socket\") pod \"e42358a7-0f5b-476a-831a-cefabde3984e\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") "
Oct 30 19:47:20.198309 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.198275    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42358a7-0f5b-476a-831a-cefabde3984e-kube-api-access-tdv4k\" (UniqueName: \"kubernetes.io/projected/e42358a7-0f5b-476a-831a-cefabde3984e-kube-api-access-tdv4k\") pod \"e42358a7-0f5b-476a-831a-cefabde3984e\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") "
Oct 30 19:47:20.198346 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.198314    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-istio-envoy\") pod \"e42358a7-0f5b-476a-831a-cefabde3984e\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") "
Oct 30 19:47:20.198407 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.198342    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e42358a7-0f5b-476a-831a-cefabde3984e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42358a7-0f5b-476a-831a-cefabde3984e-istiod-ca-cert\") pod \"e42358a7-0f5b-476a-831a-cefabde3984e\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") "
Oct 30 19:47:20.198465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.198450    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-istio-data\") pod \"e42358a7-0f5b-476a-831a-cefabde3984e\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") "
Oct 30 19:47:20.198556 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.198531    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-workload-socket" pod "e42358a7-0f5b-476a-831a-cefabde3984e" (UID: "e42358a7-0f5b-476a-831a-cefabde3984e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:20.198614 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.198540    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e42358a7-0f5b-476a-831a-cefabde3984e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42358a7-0f5b-476a-831a-cefabde3984e-istio-podinfo\") pod \"e42358a7-0f5b-476a-831a-cefabde3984e\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") "
Oct 30 19:47:20.198672 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.198634    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42358a7-0f5b-476a-831a-cefabde3984e-istio-token\" (UniqueName: \"kubernetes.io/projected/e42358a7-0f5b-476a-831a-cefabde3984e-istio-token\") pod \"e42358a7-0f5b-476a-831a-cefabde3984e\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") "
Oct 30 19:47:20.198730 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.198700    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-workload-certs\") pod \"e42358a7-0f5b-476a-831a-cefabde3984e\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") "
Oct 30 19:47:20.198803 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.198719    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-istio-data" pod "e42358a7-0f5b-476a-831a-cefabde3984e" (UID: "e42358a7-0f5b-476a-831a-cefabde3984e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:20.198803 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.198729    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42358a7-0f5b-476a-831a-cefabde3984e-istiod-ca-cert" pod "e42358a7-0f5b-476a-831a-cefabde3984e" (UID: "e42358a7-0f5b-476a-831a-cefabde3984e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:20.198803 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.198757    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-credential-socket\") pod \"e42358a7-0f5b-476a-831a-cefabde3984e\" (UID: \"e42358a7-0f5b-476a-831a-cefabde3984e\") "
Oct 30 19:47:20.198969 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.198945    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-workload-certs" pod "e42358a7-0f5b-476a-831a-cefabde3984e" (UID: "e42358a7-0f5b-476a-831a-cefabde3984e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:20.199048 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.199030    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-credential-socket" pod "e42358a7-0f5b-476a-831a-cefabde3984e" (UID: "e42358a7-0f5b-476a-831a-cefabde3984e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:20.199281 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.199266    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:20.199281 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.199282    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:20.199439 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.199291    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:20.199439 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.199300    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:20.199439 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.199310    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42358a7-0f5b-476a-831a-cefabde3984e-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:20.200451 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.200423    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42358a7-0f5b-476a-831a-cefabde3984e-kube-api-access-tdv4k" pod "e42358a7-0f5b-476a-831a-cefabde3984e" (UID: "e42358a7-0f5b-476a-831a-cefabde3984e"). InnerVolumeSpecName "kube-api-access-tdv4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:20.200547 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.200487    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-istio-envoy" pod "e42358a7-0f5b-476a-831a-cefabde3984e" (UID: "e42358a7-0f5b-476a-831a-cefabde3984e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:20.200946 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.200928    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e42358a7-0f5b-476a-831a-cefabde3984e-istio-podinfo" pod "e42358a7-0f5b-476a-831a-cefabde3984e" (UID: "e42358a7-0f5b-476a-831a-cefabde3984e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:20.201010 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.200948    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42358a7-0f5b-476a-831a-cefabde3984e-istio-token" pod "e42358a7-0f5b-476a-831a-cefabde3984e" (UID: "e42358a7-0f5b-476a-831a-cefabde3984e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:20.206134 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.206111    4712 generic.go:356] "Generic (PLEG): container finished" podID="e42358a7-0f5b-476a-831a-cefabde3984e" containerID="5bb9d130295a6d0be7045bd5a9f972f411b9201c510849a6a868f634152476c2" exitCode=0
Oct 30 19:47:20.206251 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.206156    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"
Oct 30 19:47:20.206251 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.206186    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25" event={"ID":"e42358a7-0f5b-476a-831a-cefabde3984e","Type":"ContainerDied","Data":"5bb9d130295a6d0be7045bd5a9f972f411b9201c510849a6a868f634152476c2"}
Oct 30 19:47:20.206251 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.206235    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25" event={"ID":"e42358a7-0f5b-476a-831a-cefabde3984e","Type":"ContainerDied","Data":"fb4a421533d0a87e448c660530238ab1b0439603f8d5fdfda251fc96c91d7972"}
Oct 30 19:47:20.206429 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.206258    4712 scope.go:122] "RemoveContainer" containerID="5bb9d130295a6d0be7045bd5a9f972f411b9201c510849a6a868f634152476c2"
Oct 30 19:47:20.208061 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.208042    4712 generic.go:356] "Generic (PLEG): container finished" podID="045af806-47d7-41b6-9cd7-ab0b9c851f3c" containerID="e469737daf6cc305c44bf5292b0d9161b906490879a98081f4df75c7ad4067fa" exitCode=0
Oct 30 19:47:20.208180 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.208086    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst" event={"ID":"045af806-47d7-41b6-9cd7-ab0b9c851f3c","Type":"ContainerDied","Data":"e469737daf6cc305c44bf5292b0d9161b906490879a98081f4df75c7ad4067fa"}
Oct 30 19:47:20.208180 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.208101    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"
Oct 30 19:47:20.208180 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.208117    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst" event={"ID":"045af806-47d7-41b6-9cd7-ab0b9c851f3c","Type":"ContainerDied","Data":"abe545488a61409c43591a5a267f97032ae3c213a87063e5280b0700bc8f4d4d"}
Oct 30 19:47:20.215585 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.215534    4712 scope.go:122] "RemoveContainer" containerID="5bb9d130295a6d0be7045bd5a9f972f411b9201c510849a6a868f634152476c2"
Oct 30 19:47:20.215875 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:47:20.215854    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bb9d130295a6d0be7045bd5a9f972f411b9201c510849a6a868f634152476c2\": container with ID starting with 5bb9d130295a6d0be7045bd5a9f972f411b9201c510849a6a868f634152476c2 not found: ID does not exist" containerID="5bb9d130295a6d0be7045bd5a9f972f411b9201c510849a6a868f634152476c2"
Oct 30 19:47:20.215971 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.215881    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb9d130295a6d0be7045bd5a9f972f411b9201c510849a6a868f634152476c2"} err="failed to get container status \"5bb9d130295a6d0be7045bd5a9f972f411b9201c510849a6a868f634152476c2\": rpc error: code = NotFound desc = could not find container \"5bb9d130295a6d0be7045bd5a9f972f411b9201c510849a6a868f634152476c2\": container with ID starting with 5bb9d130295a6d0be7045bd5a9f972f411b9201c510849a6a868f634152476c2 not found: ID does not exist"
Oct 30 19:47:20.215971 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.215897    4712 scope.go:122] "RemoveContainer" containerID="e469737daf6cc305c44bf5292b0d9161b906490879a98081f4df75c7ad4067fa"
Oct 30 19:47:20.225940 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.225922    4712 scope.go:122] "RemoveContainer" containerID="d4d13207653fbc31f850ff0f17e0e35840f4488515bdad29a82d1b7357365a07"
Oct 30 19:47:20.236170 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.236142    4712 scope.go:122] "RemoveContainer" containerID="e469737daf6cc305c44bf5292b0d9161b906490879a98081f4df75c7ad4067fa"
Oct 30 19:47:20.236451 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:47:20.236433    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e469737daf6cc305c44bf5292b0d9161b906490879a98081f4df75c7ad4067fa\": container with ID starting with e469737daf6cc305c44bf5292b0d9161b906490879a98081f4df75c7ad4067fa not found: ID does not exist" containerID="e469737daf6cc305c44bf5292b0d9161b906490879a98081f4df75c7ad4067fa"
Oct 30 19:47:20.236546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.236462    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e469737daf6cc305c44bf5292b0d9161b906490879a98081f4df75c7ad4067fa"} err="failed to get container status \"e469737daf6cc305c44bf5292b0d9161b906490879a98081f4df75c7ad4067fa\": rpc error: code = NotFound desc = could not find container \"e469737daf6cc305c44bf5292b0d9161b906490879a98081f4df75c7ad4067fa\": container with ID starting with e469737daf6cc305c44bf5292b0d9161b906490879a98081f4df75c7ad4067fa not found: ID does not exist"
Oct 30 19:47:20.236546 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.236483    4712 scope.go:122] "RemoveContainer" containerID="d4d13207653fbc31f850ff0f17e0e35840f4488515bdad29a82d1b7357365a07"
Oct 30 19:47:20.236788 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:47:20.236766    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d13207653fbc31f850ff0f17e0e35840f4488515bdad29a82d1b7357365a07\": container with ID starting with d4d13207653fbc31f850ff0f17e0e35840f4488515bdad29a82d1b7357365a07 not found: ID does not exist" containerID="d4d13207653fbc31f850ff0f17e0e35840f4488515bdad29a82d1b7357365a07"
Oct 30 19:47:20.236832 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.236793    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d13207653fbc31f850ff0f17e0e35840f4488515bdad29a82d1b7357365a07"} err="failed to get container status \"d4d13207653fbc31f850ff0f17e0e35840f4488515bdad29a82d1b7357365a07\": rpc error: code = NotFound desc = could not find container \"d4d13207653fbc31f850ff0f17e0e35840f4488515bdad29a82d1b7357365a07\": container with ID starting with d4d13207653fbc31f850ff0f17e0e35840f4488515bdad29a82d1b7357365a07 not found: ID does not exist"
Oct 30 19:47:20.299837 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.299806    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"]
Oct 30 19:47:20.300334 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.300206    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-credential-socket\") pod \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") "
Oct 30 19:47:20.300334 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.300273    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-podinfo\") pod \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") "
Oct 30 19:47:20.300334 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.300308    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-workload-certs\") pod \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") "
Oct 30 19:47:20.300563 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.300337    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-token\" (UniqueName: \"kubernetes.io/projected/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-token\") pod \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") "
Oct 30 19:47:20.300563 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.300403    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-envoy\") pod \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") "
Oct 30 19:47:20.300563 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.300460    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-workload-socket\") pod \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") "
Oct 30 19:47:20.300563 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.300534    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045af806-47d7-41b6-9cd7-ab0b9c851f3c-kube-api-access-hzcbq\" (UniqueName: \"kubernetes.io/projected/045af806-47d7-41b6-9cd7-ab0b9c851f3c-kube-api-access-hzcbq\") pod \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") "
Oct 30 19:47:20.300777 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.300600    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-data\") pod \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") "
Oct 30 19:47:20.300777 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.300619    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-credential-socket" pod "045af806-47d7-41b6-9cd7-ab0b9c851f3c" (UID: "045af806-47d7-41b6-9cd7-ab0b9c851f3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:20.300777 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.300629    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istiod-ca-cert\") pod \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\" (UID: \"045af806-47d7-41b6-9cd7-ab0b9c851f3c\") "
Oct 30 19:47:20.300923 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.300870    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istiod-ca-cert" pod "045af806-47d7-41b6-9cd7-ab0b9c851f3c" (UID: "045af806-47d7-41b6-9cd7-ab0b9c851f3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:20.302040 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.301194    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-workload-certs" pod "045af806-47d7-41b6-9cd7-ab0b9c851f3c" (UID: "045af806-47d7-41b6-9cd7-ab0b9c851f3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:20.302040 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.301262    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-workload-socket" pod "045af806-47d7-41b6-9cd7-ab0b9c851f3c" (UID: "045af806-47d7-41b6-9cd7-ab0b9c851f3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:20.302040 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.301278    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42358a7-0f5b-476a-831a-cefabde3984e-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:20.302040 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.301322    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42358a7-0f5b-476a-831a-cefabde3984e-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:20.302040 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.301337    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:20.302040 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.301351    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdv4k\" (UniqueName: \"kubernetes.io/projected/e42358a7-0f5b-476a-831a-cefabde3984e-kube-api-access-tdv4k\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:20.302040 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.301394    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42358a7-0f5b-476a-831a-cefabde3984e-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:20.302040 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.301409    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:20.302040 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.301502    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-data" pod "045af806-47d7-41b6-9cd7-ab0b9c851f3c" (UID: "045af806-47d7-41b6-9cd7-ab0b9c851f3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:20.303089 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.303062    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-podinfo" pod "045af806-47d7-41b6-9cd7-ab0b9c851f3c" (UID: "045af806-47d7-41b6-9cd7-ab0b9c851f3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:20.303191 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.303148    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-envoy" pod "045af806-47d7-41b6-9cd7-ab0b9c851f3c" (UID: "045af806-47d7-41b6-9cd7-ab0b9c851f3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:20.303433 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.303414    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-token" pod "045af806-47d7-41b6-9cd7-ab0b9c851f3c" (UID: "045af806-47d7-41b6-9cd7-ab0b9c851f3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:20.303517 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.303494    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045af806-47d7-41b6-9cd7-ab0b9c851f3c-kube-api-access-hzcbq" pod "045af806-47d7-41b6-9cd7-ab0b9c851f3c" (UID: "045af806-47d7-41b6-9cd7-ab0b9c851f3c"). InnerVolumeSpecName "kube-api-access-hzcbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:20.304545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.304524    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-84n25"]
Oct 30 19:47:20.402825 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.402797    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:20.402825 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.402822    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:20.402825 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.402831    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:20.403020 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.402839    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:20.403020 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.402848    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:20.403020 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.402857    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzcbq\" (UniqueName: \"kubernetes.io/projected/045af806-47d7-41b6-9cd7-ab0b9c851f3c-kube-api-access-hzcbq\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:20.403020 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.402865    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/045af806-47d7-41b6-9cd7-ab0b9c851f3c-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:20.555545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.555475    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"]
Oct 30 19:47:20.566887 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.566851    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b4c34-openshift-default-7bfcb4b557-nwcst"]
Oct 30 19:47:20.851789 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.851721    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="045af806-47d7-41b6-9cd7-ab0b9c851f3c" path="/var/lib/kubelet/pods/045af806-47d7-41b6-9cd7-ab0b9c851f3c/volumes"
Oct 30 19:47:20.852327 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:20.852307    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e42358a7-0f5b-476a-831a-cefabde3984e" path="/var/lib/kubelet/pods/e42358a7-0f5b-476a-831a-cefabde3984e/volumes"
Oct 30 19:47:21.130774 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:21.130740    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:21.188689 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:21.188663    4712 patch_prober.go:29] interesting pod/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:47:21.188808 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:21.188725    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp" podUID="3fce9cac-38e0-436f-a4aa-4f82b908e11e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:47:21.673160 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:21.673127    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:21.829337 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:21.829303    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:22.287771 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:22.287735    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:23.229811 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.229782    4712 generic.go:356] "Generic (PLEG): container finished" podID="3fce9cac-38e0-436f-a4aa-4f82b908e11e" containerID="1fd0a1b7ca423ef3634ec9d8ae6a6f7ee03cdaf34f63a430faee2f5bc16da024" exitCode=0
Oct 30 19:47:23.230172 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.229854    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp" event={"ID":"3fce9cac-38e0-436f-a4aa-4f82b908e11e","Type":"ContainerDied","Data":"1fd0a1b7ca423ef3634ec9d8ae6a6f7ee03cdaf34f63a430faee2f5bc16da024"}
Oct 30 19:47:23.230172 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.229913    4712 scope.go:122] "RemoveContainer" containerID="a2d80302043fb98ff8f87b011a770cfa15237d7ea139100c7a7ef665859d1421"
Oct 30 19:47:23.270558 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.270506    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:47:23.333035 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.333012    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-envoy\") pod \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") "
Oct 30 19:47:23.333162 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.333043    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-podinfo\") pod \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") "
Oct 30 19:47:23.333162 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.333078    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-data\") pod \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") "
Oct 30 19:47:23.333244 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.333195    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-token\" (UniqueName: \"kubernetes.io/projected/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-token\") pod \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") "
Oct 30 19:47:23.333294 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.333264    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-credential-socket\") pod \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") "
Oct 30 19:47:23.333328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.333300    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-data" pod "3fce9cac-38e0-436f-a4aa-4f82b908e11e" (UID: "3fce9cac-38e0-436f-a4aa-4f82b908e11e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:23.333406 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.333339    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-workload-socket\") pod \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") "
Oct 30 19:47:23.333474 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.333422    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fce9cac-38e0-436f-a4aa-4f82b908e11e-kube-api-access-t6cz7\" (UniqueName: \"kubernetes.io/projected/3fce9cac-38e0-436f-a4aa-4f82b908e11e-kube-api-access-t6cz7\") pod \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") "
Oct 30 19:47:23.333526 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.333482    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istiod-ca-cert\") pod \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") "
Oct 30 19:47:23.333576 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.333531    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-workload-certs\") pod \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\" (UID: \"3fce9cac-38e0-436f-a4aa-4f82b908e11e\") "
Oct 30 19:47:23.333630 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.333573    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-credential-socket" pod "3fce9cac-38e0-436f-a4aa-4f82b908e11e" (UID: "3fce9cac-38e0-436f-a4aa-4f82b908e11e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:23.333848 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.333767    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-workload-socket" pod "3fce9cac-38e0-436f-a4aa-4f82b908e11e" (UID: "3fce9cac-38e0-436f-a4aa-4f82b908e11e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:23.333848 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.333818    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-workload-certs" pod "3fce9cac-38e0-436f-a4aa-4f82b908e11e" (UID: "3fce9cac-38e0-436f-a4aa-4f82b908e11e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:23.334001 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.333843    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istiod-ca-cert" pod "3fce9cac-38e0-436f-a4aa-4f82b908e11e" (UID: "3fce9cac-38e0-436f-a4aa-4f82b908e11e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:23.334130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.334113    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:23.334195 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.334138    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:23.334195 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.334155    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:23.334195 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.334169    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:23.334195 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.334182    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:23.335202 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.335182    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-envoy" pod "3fce9cac-38e0-436f-a4aa-4f82b908e11e" (UID: "3fce9cac-38e0-436f-a4aa-4f82b908e11e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:23.335443 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.335420    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-podinfo" pod "3fce9cac-38e0-436f-a4aa-4f82b908e11e" (UID: "3fce9cac-38e0-436f-a4aa-4f82b908e11e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:23.335662 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.335637    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-token" pod "3fce9cac-38e0-436f-a4aa-4f82b908e11e" (UID: "3fce9cac-38e0-436f-a4aa-4f82b908e11e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:23.335662 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.335656    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fce9cac-38e0-436f-a4aa-4f82b908e11e-kube-api-access-t6cz7" pod "3fce9cac-38e0-436f-a4aa-4f82b908e11e" (UID: "3fce9cac-38e0-436f-a4aa-4f82b908e11e"). InnerVolumeSpecName "kube-api-access-t6cz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:23.434797 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.434767    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:23.434797 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.434795    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:23.434797 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.434804    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fce9cac-38e0-436f-a4aa-4f82b908e11e-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:23.435013 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.434813    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6cz7\" (UniqueName: \"kubernetes.io/projected/3fce9cac-38e0-436f-a4aa-4f82b908e11e-kube-api-access-t6cz7\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:23.579503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:23.579419    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:24.236584 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:24.236558    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"
Oct 30 19:47:24.236584 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:24.236580    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp" event={"ID":"3fce9cac-38e0-436f-a4aa-4f82b908e11e","Type":"ContainerDied","Data":"c2acd6360ee2b6eb5983b181dedaaf78673a042f903f99f8e7c1da238e6e49f6"}
Oct 30 19:47:24.237056 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:24.236631    4712 scope.go:122] "RemoveContainer" containerID="1fd0a1b7ca423ef3634ec9d8ae6a6f7ee03cdaf34f63a430faee2f5bc16da024"
Oct 30 19:47:24.300890 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:24.300867    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"]
Oct 30 19:47:24.313070 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:24.313045    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-k4vvp"]
Oct 30 19:47:24.852610 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:24.852576    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fce9cac-38e0-436f-a4aa-4f82b908e11e" path="/var/lib/kubelet/pods/3fce9cac-38e0-436f-a4aa-4f82b908e11e/volumes"
Oct 30 19:47:24.985743 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:24.985715    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59"]
Oct 30 19:47:25.036184 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:25.036156    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr"]
Oct 30 19:47:25.131402 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:25.131354    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:25.332399 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:25.332332    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:26.813537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:26.813479    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:27.130688 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:27.130655    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:27.327188 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:27.327159    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:27.400832 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:27.400755    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"]
Oct 30 19:47:27.400984 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:27.400968    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd" podUID="700a4ae0-505b-4926-93c0-ee5b5f627030" containerName="istio-proxy" containerID="cri-o://2b83080aabe07b80575808940d4a6d0b94d856768a576eeedc903a69eed8c88f" gracePeriod=30
Oct 30 19:47:27.532206 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:27.532155    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:28.424571 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:28.424526    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:29.986038 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:29.986002    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59" podUID="fd990c24-235b-4f45-92a1-da3e7c443153" containerName="payload-processing" containerID="cri-o://d0f7dcf6c92940f10ae3c0c4ec262df3e7ec3b818eed42d26f606a0dff54cf33" gracePeriod=25
Oct 30 19:47:30.037065 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:30.037030    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr" podUID="245c3fb3-0b83-4d71-be3c-81d6cafb639c" containerName="payload-pre-processing" containerID="cri-o://4aab104c581d69119fa60e244a86bebc65547f7a455fbf9663d95a6b87749a2f" gracePeriod=25
Oct 30 19:47:30.081601 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:30.081509    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:30.390233 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:30.390199    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:30.565524 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:30.565486    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:30.754163 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:30.754013    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:30.956449 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:30.956406    4712 patch_prober.go:29] interesting pod/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:47:30.956632 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:30.956490    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd" podUID="700a4ae0-505b-4926-93c0-ee5b5f627030" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:47:31.223106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:31.223065    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:31.803964 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:31.803930    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:32.644501 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.644478    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:47:32.727181 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.727118    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-token\" (UniqueName: \"kubernetes.io/projected/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-token\") pod \"700a4ae0-505b-4926-93c0-ee5b5f627030\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") "
Oct 30 19:47:32.727181 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.727177    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700a4ae0-505b-4926-93c0-ee5b5f627030-kube-api-access-rw7m6\" (UniqueName: \"kubernetes.io/projected/700a4ae0-505b-4926-93c0-ee5b5f627030-kube-api-access-rw7m6\") pod \"700a4ae0-505b-4926-93c0-ee5b5f627030\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") "
Oct 30 19:47:32.727396 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.727291    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-envoy\") pod \"700a4ae0-505b-4926-93c0-ee5b5f627030\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") "
Oct 30 19:47:32.727396 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.727342    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-workload-socket\") pod \"700a4ae0-505b-4926-93c0-ee5b5f627030\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") "
Oct 30 19:47:32.727515 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.727406    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-podinfo\") pod \"700a4ae0-505b-4926-93c0-ee5b5f627030\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") "
Oct 30 19:47:32.727515 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.727502    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/700a4ae0-505b-4926-93c0-ee5b5f627030-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700a4ae0-505b-4926-93c0-ee5b5f627030-istiod-ca-cert\") pod \"700a4ae0-505b-4926-93c0-ee5b5f627030\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") "
Oct 30 19:47:32.728060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.727618    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-workload-certs\") pod \"700a4ae0-505b-4926-93c0-ee5b5f627030\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") "
Oct 30 19:47:32.728060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.727668    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-credential-socket\") pod \"700a4ae0-505b-4926-93c0-ee5b5f627030\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") "
Oct 30 19:47:32.728060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.727696    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-data\") pod \"700a4ae0-505b-4926-93c0-ee5b5f627030\" (UID: \"700a4ae0-505b-4926-93c0-ee5b5f627030\") "
Oct 30 19:47:32.728060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.727608    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-workload-socket" pod "700a4ae0-505b-4926-93c0-ee5b5f627030" (UID: "700a4ae0-505b-4926-93c0-ee5b5f627030"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:32.728060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.727845    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-workload-certs" pod "700a4ae0-505b-4926-93c0-ee5b5f627030" (UID: "700a4ae0-505b-4926-93c0-ee5b5f627030"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:32.728060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.727918    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700a4ae0-505b-4926-93c0-ee5b5f627030-istiod-ca-cert" pod "700a4ae0-505b-4926-93c0-ee5b5f627030" (UID: "700a4ae0-505b-4926-93c0-ee5b5f627030"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:32.728377 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.728092    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-credential-socket" pod "700a4ae0-505b-4926-93c0-ee5b5f627030" (UID: "700a4ae0-505b-4926-93c0-ee5b5f627030"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:32.728377 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.728261    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/700a4ae0-505b-4926-93c0-ee5b5f627030-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:32.728377 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.728282    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:32.728377 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.728296    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:32.728377 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.728311    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:32.728642 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.728609    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-data" pod "700a4ae0-505b-4926-93c0-ee5b5f627030" (UID: "700a4ae0-505b-4926-93c0-ee5b5f627030"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:32.730163 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.730137    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700a4ae0-505b-4926-93c0-ee5b5f627030-kube-api-access-rw7m6" pod "700a4ae0-505b-4926-93c0-ee5b5f627030" (UID: "700a4ae0-505b-4926-93c0-ee5b5f627030"). InnerVolumeSpecName "kube-api-access-rw7m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:32.731561 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.731538    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-envoy" pod "700a4ae0-505b-4926-93c0-ee5b5f627030" (UID: "700a4ae0-505b-4926-93c0-ee5b5f627030"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:32.732483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.732461    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-token" pod "700a4ae0-505b-4926-93c0-ee5b5f627030" (UID: "700a4ae0-505b-4926-93c0-ee5b5f627030"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:32.732844 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.732822    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-podinfo" pod "700a4ae0-505b-4926-93c0-ee5b5f627030" (UID: "700a4ae0-505b-4926-93c0-ee5b5f627030"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:47:32.829567 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.829524    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:32.829567 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.829550    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:32.829567 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.829564    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw7m6\" (UniqueName: \"kubernetes.io/projected/700a4ae0-505b-4926-93c0-ee5b5f627030-kube-api-access-rw7m6\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:32.829787 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.829576    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:32.829787 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:32.829589    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/700a4ae0-505b-4926-93c0-ee5b5f627030-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:33.300678 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:33.300643    4712 generic.go:356] "Generic (PLEG): container finished" podID="700a4ae0-505b-4926-93c0-ee5b5f627030" containerID="2b83080aabe07b80575808940d4a6d0b94d856768a576eeedc903a69eed8c88f" exitCode=0
Oct 30 19:47:33.300825 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:33.300686    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"
Oct 30 19:47:33.300825 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:33.300728    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd" event={"ID":"700a4ae0-505b-4926-93c0-ee5b5f627030","Type":"ContainerDied","Data":"2b83080aabe07b80575808940d4a6d0b94d856768a576eeedc903a69eed8c88f"}
Oct 30 19:47:33.300825 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:33.300781    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd" event={"ID":"700a4ae0-505b-4926-93c0-ee5b5f627030","Type":"ContainerDied","Data":"4335efbc25ff09add8c5e3460366e4a7c562bb6538bcc19db58a20d62e9bb8bd"}
Oct 30 19:47:33.300825 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:33.300802    4712 scope.go:122] "RemoveContainer" containerID="2b83080aabe07b80575808940d4a6d0b94d856768a576eeedc903a69eed8c88f"
Oct 30 19:47:33.310169 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:33.310150    4712 scope.go:122] "RemoveContainer" containerID="2b83080aabe07b80575808940d4a6d0b94d856768a576eeedc903a69eed8c88f"
Oct 30 19:47:33.310467 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:47:33.310445    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b83080aabe07b80575808940d4a6d0b94d856768a576eeedc903a69eed8c88f\": container with ID starting with 2b83080aabe07b80575808940d4a6d0b94d856768a576eeedc903a69eed8c88f not found: ID does not exist" containerID="2b83080aabe07b80575808940d4a6d0b94d856768a576eeedc903a69eed8c88f"
Oct 30 19:47:33.310524 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:33.310477    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b83080aabe07b80575808940d4a6d0b94d856768a576eeedc903a69eed8c88f"} err="failed to get container status \"2b83080aabe07b80575808940d4a6d0b94d856768a576eeedc903a69eed8c88f\": rpc error: code = NotFound desc = could not find container \"2b83080aabe07b80575808940d4a6d0b94d856768a576eeedc903a69eed8c88f\": container with ID starting with 2b83080aabe07b80575808940d4a6d0b94d856768a576eeedc903a69eed8c88f not found: ID does not exist"
Oct 30 19:47:33.322263 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:33.322239    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:33.354197 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:33.354174    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"]
Oct 30 19:47:33.366193 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:33.366162    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-67953efd-openshift-default-f666c5445-q8lpd"]
Oct 30 19:47:33.635642 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:33.635612    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:34.269243 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.269190    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main" containerID="cri-o://3979c4a006abcd517ebee57e8953feae98b86d3ecd048830bc7569f4a745c51a" gracePeriod=45
Oct 30 19:47:34.553942 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.553912    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:47:34.647933 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.647894    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-dshm\") pod \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") "
Oct 30 19:47:34.648115 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.648008    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-home\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-home\") pod \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") "
Oct 30 19:47:34.648115 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.648051    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-tls-certs\") pod \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") "
Oct 30 19:47:34.648246 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.648140    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-model-cache\") pod \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") "
Oct 30 19:47:34.648246 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.648180    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-kube-api-access-fp42w\" (UniqueName: \"kubernetes.io/projected/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-kube-api-access-fp42w\") pod \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") "
Oct 30 19:47:34.648380 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.648257    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-tmp-dir\") pod \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\" (UID: \"5e3a8512-59ff-45bf-b7c7-55369c05d8ce\") "
Oct 30 19:47:34.649462 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.649430    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-model-cache" pod "5e3a8512-59ff-45bf-b7c7-55369c05d8ce" (UID: "5e3a8512-59ff-45bf-b7c7-55369c05d8ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:34.649690 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.649666    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-home" pod "5e3a8512-59ff-45bf-b7c7-55369c05d8ce" (UID: "5e3a8512-59ff-45bf-b7c7-55369c05d8ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:34.649775 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.649758    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-tmp-dir" pod "5e3a8512-59ff-45bf-b7c7-55369c05d8ce" (UID: "5e3a8512-59ff-45bf-b7c7-55369c05d8ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:34.652879 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.652854    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-tls-certs" pod "5e3a8512-59ff-45bf-b7c7-55369c05d8ce" (UID: "5e3a8512-59ff-45bf-b7c7-55369c05d8ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:34.662384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.662188    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-kube-api-access-fp42w" pod "5e3a8512-59ff-45bf-b7c7-55369c05d8ce" (UID: "5e3a8512-59ff-45bf-b7c7-55369c05d8ce"). InnerVolumeSpecName "kube-api-access-fp42w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:34.665393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.665247    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-dshm" pod "5e3a8512-59ff-45bf-b7c7-55369c05d8ce" (UID: "5e3a8512-59ff-45bf-b7c7-55369c05d8ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:34.763130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.749498    4712 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-tmp-dir\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:34.763130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.749534    4712 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-dshm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:34.763130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.749547    4712 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-home\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:34.763130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.749559    4712 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-tls-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:34.763130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.749573    4712 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-model-cache\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:34.763130 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.749586    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp42w\" (UniqueName: \"kubernetes.io/projected/5e3a8512-59ff-45bf-b7c7-55369c05d8ce-kube-api-access-fp42w\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:34.862991 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:34.861620    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="700a4ae0-505b-4926-93c0-ee5b5f627030" path="/var/lib/kubelet/pods/700a4ae0-505b-4926-93c0-ee5b5f627030/volumes"
Oct 30 19:47:35.345676 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:35.345641    4712 generic.go:356] "Generic (PLEG): container finished" podID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerID="3979c4a006abcd517ebee57e8953feae98b86d3ecd048830bc7569f4a745c51a" exitCode=0
Oct 30 19:47:35.350454 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:35.350422    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" event={"ID":"5e3a8512-59ff-45bf-b7c7-55369c05d8ce","Type":"ContainerDied","Data":"3979c4a006abcd517ebee57e8953feae98b86d3ecd048830bc7569f4a745c51a"}
Oct 30 19:47:35.350643 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:35.350625    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc" event={"ID":"5e3a8512-59ff-45bf-b7c7-55369c05d8ce","Type":"ContainerDied","Data":"0666501b4bc96f8570097fb7c1e5ff15c87f55b9c7686caaef54b0d52946f06c"}
Oct 30 19:47:35.350769 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:35.350752    4712 scope.go:122] "RemoveContainer" containerID="3979c4a006abcd517ebee57e8953feae98b86d3ecd048830bc7569f4a745c51a"
Oct 30 19:47:35.351031 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:35.351013    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"
Oct 30 19:47:35.394867 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:35.394772    4712 scope.go:122] "RemoveContainer" containerID="3979c4a006abcd517ebee57e8953feae98b86d3ecd048830bc7569f4a745c51a"
Oct 30 19:47:35.395334 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:47:35.395310    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3979c4a006abcd517ebee57e8953feae98b86d3ecd048830bc7569f4a745c51a\": container with ID starting with 3979c4a006abcd517ebee57e8953feae98b86d3ecd048830bc7569f4a745c51a not found: ID does not exist" containerID="3979c4a006abcd517ebee57e8953feae98b86d3ecd048830bc7569f4a745c51a"
Oct 30 19:47:35.395661 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:35.395344    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3979c4a006abcd517ebee57e8953feae98b86d3ecd048830bc7569f4a745c51a"} err="failed to get container status \"3979c4a006abcd517ebee57e8953feae98b86d3ecd048830bc7569f4a745c51a\": rpc error: code = NotFound desc = could not find container \"3979c4a006abcd517ebee57e8953feae98b86d3ecd048830bc7569f4a745c51a\": container with ID starting with 3979c4a006abcd517ebee57e8953feae98b86d3ecd048830bc7569f4a745c51a not found: ID does not exist"
Oct 30 19:47:35.449444 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:35.449413    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"]
Oct 30 19:47:35.468994 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:35.468963    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f913c6/distinct-w4-f913c6-kserve-7767c75f85-7pghc"]
Oct 30 19:47:35.488211 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:35.488183    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:36.146810 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:36.146771    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"]
Oct 30 19:47:36.162826 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:36.162795    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"]
Oct 30 19:47:36.333395 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:36.327734    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:36.401038 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:47:36.400961    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:47:36.855624 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:36.855525    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" path="/var/lib/kubelet/pods/5e3a8512-59ff-45bf-b7c7-55369c05d8ce/volumes"
Oct 30 19:47:37.660438 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:37.660408    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:38.189224 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:38.189189    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59" podUID="fd990c24-235b-4f45-92a1-da3e7c443153" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:47:38.189224 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:38.189216    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr" podUID="245c3fb3-0b83-4d71-be3c-81d6cafb639c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:47:39.571752 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:39.571704    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:41.188736 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:41.188678    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:41.950627 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:41.950596    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c"]
Oct 30 19:47:42.139757 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:42.139721    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:42.295225 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:42.295154    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:42.893059 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:42.893030    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:44.357307 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:44.357253    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:45.849744 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:45.849708    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:46.375817 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:46.375782    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:46.548471 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:46.548431    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:46.950907 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:46.950875    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c" podUID="d19408be-2ea5-4445-99a5-9094c497b95f" containerName="payload-processing" containerID="cri-o://6122c058157a02ff805345e35e90d1c10ab70a725c5d40017f7d6d400815d43d" gracePeriod=25
Oct 30 19:47:46.984980 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:46.984953    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:48.189271 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:48.189185    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59" podUID="fd990c24-235b-4f45-92a1-da3e7c443153" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:47:48.189690 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:48.189196    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr" podUID="245c3fb3-0b83-4d71-be3c-81d6cafb639c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:47:48.556599 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:48.556521    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:50.152717 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.152685    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:50.288291 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.288267    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59"
Oct 30 19:47:50.311298 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.311276    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr"
Oct 30 19:47:50.323462 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.323440    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd990c24-235b-4f45-92a1-da3e7c443153-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd990c24-235b-4f45-92a1-da3e7c443153-plugins-config-volume\") pod \"fd990c24-235b-4f45-92a1-da3e7c443153\" (UID: \"fd990c24-235b-4f45-92a1-da3e7c443153\") "
Oct 30 19:47:50.323578 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.323568    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd990c24-235b-4f45-92a1-da3e7c443153-kube-api-access-dcrkn\" (UniqueName: \"kubernetes.io/projected/fd990c24-235b-4f45-92a1-da3e7c443153-kube-api-access-dcrkn\") pod \"fd990c24-235b-4f45-92a1-da3e7c443153\" (UID: \"fd990c24-235b-4f45-92a1-da3e7c443153\") "
Oct 30 19:47:50.323761 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.323739    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd990c24-235b-4f45-92a1-da3e7c443153-plugins-config-volume" pod "fd990c24-235b-4f45-92a1-da3e7c443153" (UID: "fd990c24-235b-4f45-92a1-da3e7c443153"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:50.323872 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.323859    4712 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd990c24-235b-4f45-92a1-da3e7c443153-plugins-config-volume\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:50.325460 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.325433    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd990c24-235b-4f45-92a1-da3e7c443153-kube-api-access-dcrkn" pod "fd990c24-235b-4f45-92a1-da3e7c443153" (UID: "fd990c24-235b-4f45-92a1-da3e7c443153"). InnerVolumeSpecName "kube-api-access-dcrkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:50.424457 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.424426    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/245c3fb3-0b83-4d71-be3c-81d6cafb639c-kube-api-access-4g2hz\" (UniqueName: \"kubernetes.io/projected/245c3fb3-0b83-4d71-be3c-81d6cafb639c-kube-api-access-4g2hz\") pod \"245c3fb3-0b83-4d71-be3c-81d6cafb639c\" (UID: \"245c3fb3-0b83-4d71-be3c-81d6cafb639c\") "
Oct 30 19:47:50.424630 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.424485    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/245c3fb3-0b83-4d71-be3c-81d6cafb639c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/245c3fb3-0b83-4d71-be3c-81d6cafb639c-plugins-config-volume\") pod \"245c3fb3-0b83-4d71-be3c-81d6cafb639c\" (UID: \"245c3fb3-0b83-4d71-be3c-81d6cafb639c\") "
Oct 30 19:47:50.424731 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.424718    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcrkn\" (UniqueName: \"kubernetes.io/projected/fd990c24-235b-4f45-92a1-da3e7c443153-kube-api-access-dcrkn\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:50.424895 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.424872    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245c3fb3-0b83-4d71-be3c-81d6cafb639c-plugins-config-volume" pod "245c3fb3-0b83-4d71-be3c-81d6cafb639c" (UID: "245c3fb3-0b83-4d71-be3c-81d6cafb639c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:50.426397 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.426378    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245c3fb3-0b83-4d71-be3c-81d6cafb639c-kube-api-access-4g2hz" pod "245c3fb3-0b83-4d71-be3c-81d6cafb639c" (UID: "245c3fb3-0b83-4d71-be3c-81d6cafb639c"). InnerVolumeSpecName "kube-api-access-4g2hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:50.462190 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.462160    4712 generic.go:356] "Generic (PLEG): container finished" podID="245c3fb3-0b83-4d71-be3c-81d6cafb639c" containerID="4aab104c581d69119fa60e244a86bebc65547f7a455fbf9663d95a6b87749a2f" exitCode=0
Oct 30 19:47:50.462308 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.462213    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr"
Oct 30 19:47:50.462308 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.462244    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr" event={"ID":"245c3fb3-0b83-4d71-be3c-81d6cafb639c","Type":"ContainerDied","Data":"4aab104c581d69119fa60e244a86bebc65547f7a455fbf9663d95a6b87749a2f"}
Oct 30 19:47:50.462308 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.462294    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr" event={"ID":"245c3fb3-0b83-4d71-be3c-81d6cafb639c","Type":"ContainerDied","Data":"4983e00ca5220ccd2eb84e78e25dc5dd82a357e6c05dc54080422b2c14fea5e5"}
Oct 30 19:47:50.462506 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.462316    4712 scope.go:122] "RemoveContainer" containerID="4aab104c581d69119fa60e244a86bebc65547f7a455fbf9663d95a6b87749a2f"
Oct 30 19:47:50.463823 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.463796    4712 generic.go:356] "Generic (PLEG): container finished" podID="fd990c24-235b-4f45-92a1-da3e7c443153" containerID="d0f7dcf6c92940f10ae3c0c4ec262df3e7ec3b818eed42d26f606a0dff54cf33" exitCode=0
Oct 30 19:47:50.463933 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.463830    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59"
Oct 30 19:47:50.463933 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.463868    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59" event={"ID":"fd990c24-235b-4f45-92a1-da3e7c443153","Type":"ContainerDied","Data":"d0f7dcf6c92940f10ae3c0c4ec262df3e7ec3b818eed42d26f606a0dff54cf33"}
Oct 30 19:47:50.463933 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.463897    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59" event={"ID":"fd990c24-235b-4f45-92a1-da3e7c443153","Type":"ContainerDied","Data":"f50b33c317382b3da5ab25863d95fdac0bf9bd5121ea0bb4450eb64340c3f1fa"}
Oct 30 19:47:50.471669 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.471651    4712 scope.go:122] "RemoveContainer" containerID="4aab104c581d69119fa60e244a86bebc65547f7a455fbf9663d95a6b87749a2f"
Oct 30 19:47:50.471954 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:47:50.471931    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aab104c581d69119fa60e244a86bebc65547f7a455fbf9663d95a6b87749a2f\": container with ID starting with 4aab104c581d69119fa60e244a86bebc65547f7a455fbf9663d95a6b87749a2f not found: ID does not exist" containerID="4aab104c581d69119fa60e244a86bebc65547f7a455fbf9663d95a6b87749a2f"
Oct 30 19:47:50.472009 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.471968    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aab104c581d69119fa60e244a86bebc65547f7a455fbf9663d95a6b87749a2f"} err="failed to get container status \"4aab104c581d69119fa60e244a86bebc65547f7a455fbf9663d95a6b87749a2f\": rpc error: code = NotFound desc = could not find container \"4aab104c581d69119fa60e244a86bebc65547f7a455fbf9663d95a6b87749a2f\": container with ID starting with 4aab104c581d69119fa60e244a86bebc65547f7a455fbf9663d95a6b87749a2f not found: ID does not exist"
Oct 30 19:47:50.472009 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.471992    4712 scope.go:122] "RemoveContainer" containerID="d0f7dcf6c92940f10ae3c0c4ec262df3e7ec3b818eed42d26f606a0dff54cf33"
Oct 30 19:47:50.481900 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.481880    4712 scope.go:122] "RemoveContainer" containerID="d0f7dcf6c92940f10ae3c0c4ec262df3e7ec3b818eed42d26f606a0dff54cf33"
Oct 30 19:47:50.482323 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:47:50.482302    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f7dcf6c92940f10ae3c0c4ec262df3e7ec3b818eed42d26f606a0dff54cf33\": container with ID starting with d0f7dcf6c92940f10ae3c0c4ec262df3e7ec3b818eed42d26f606a0dff54cf33 not found: ID does not exist" containerID="d0f7dcf6c92940f10ae3c0c4ec262df3e7ec3b818eed42d26f606a0dff54cf33"
Oct 30 19:47:50.482426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.482339    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f7dcf6c92940f10ae3c0c4ec262df3e7ec3b818eed42d26f606a0dff54cf33"} err="failed to get container status \"d0f7dcf6c92940f10ae3c0c4ec262df3e7ec3b818eed42d26f606a0dff54cf33\": rpc error: code = NotFound desc = could not find container \"d0f7dcf6c92940f10ae3c0c4ec262df3e7ec3b818eed42d26f606a0dff54cf33\": container with ID starting with d0f7dcf6c92940f10ae3c0c4ec262df3e7ec3b818eed42d26f606a0dff54cf33 not found: ID does not exist"
Oct 30 19:47:50.520295 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.520239    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59"]
Oct 30 19:47:50.522167 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.522149    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f913c6-76d6f8fb7d-hjl59"]
Oct 30 19:47:50.525328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.525313    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g2hz\" (UniqueName: \"kubernetes.io/projected/245c3fb3-0b83-4d71-be3c-81d6cafb639c-kube-api-access-4g2hz\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:50.525404 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.525331    4712 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/245c3fb3-0b83-4d71-be3c-81d6cafb639c-plugins-config-volume\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:50.555599 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.555578    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr"]
Oct 30 19:47:50.560134 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.560112    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f913c6-5457ccff55-t7qbr"]
Oct 30 19:47:50.851802 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.851720    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="245c3fb3-0b83-4d71-be3c-81d6cafb639c" path="/var/lib/kubelet/pods/245c3fb3-0b83-4d71-be3c-81d6cafb639c/volumes"
Oct 30 19:47:50.852227 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:50.852209    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd990c24-235b-4f45-92a1-da3e7c443153" path="/var/lib/kubelet/pods/fd990c24-235b-4f45-92a1-da3e7c443153/volumes"
Oct 30 19:47:51.203931 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.203899    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" containerID="cri-o://5bf841fa8a53fbc255a62c00e33fedd2bc42b7e233c7757eba6b88a0777d61c9" gracePeriod=45
Oct 30 19:47:51.205124 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.205101    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main" containerID="cri-o://b57bb798904714d0e87194dc71c0aa595fcc5dc1f7963bfb16fef73d8614f26d" gracePeriod=45
Oct 30 19:47:51.307401 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.307338    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": dial tcp 10.133.0.66:8000: connect: connection refused"
Oct 30 19:47:51.471333 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.471306    4712 generic.go:356] "Generic (PLEG): container finished" podID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerID="5bf841fa8a53fbc255a62c00e33fedd2bc42b7e233c7757eba6b88a0777d61c9" exitCode=0
Oct 30 19:47:51.471483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.471413    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" event={"ID":"e640ac87-e1bd-4f61-8848-f82dd4835442","Type":"ContainerDied","Data":"5bf841fa8a53fbc255a62c00e33fedd2bc42b7e233c7757eba6b88a0777d61c9"}
Oct 30 19:47:51.471483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.471467    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk" event={"ID":"e640ac87-e1bd-4f61-8848-f82dd4835442","Type":"ContainerDied","Data":"42c6e57cf0d1a007ba3725270f7d17eddcf4550f138ff1056ec3692aed4ba791"}
Oct 30 19:47:51.471561 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.471487    4712 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42c6e57cf0d1a007ba3725270f7d17eddcf4550f138ff1056ec3692aed4ba791"
Oct 30 19:47:51.476704 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.476680    4712 generic.go:356] "Generic (PLEG): container finished" podID="d3bee1cf-6278-4666-93db-a483e7dca394" containerID="b57bb798904714d0e87194dc71c0aa595fcc5dc1f7963bfb16fef73d8614f26d" exitCode=0
Oct 30 19:47:51.476812 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.476743    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" event={"ID":"d3bee1cf-6278-4666-93db-a483e7dca394","Type":"ContainerDied","Data":"b57bb798904714d0e87194dc71c0aa595fcc5dc1f7963bfb16fef73d8614f26d"}
Oct 30 19:47:51.477731 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.477713    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:47:51.485134 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.485117    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:47:51.534578 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.534553    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-dshm\") pod \"e640ac87-e1bd-4f61-8848-f82dd4835442\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") "
Oct 30 19:47:51.534707 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.534594    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e640ac87-e1bd-4f61-8848-f82dd4835442-tls-certs\" (UniqueName: \"kubernetes.io/secret/e640ac87-e1bd-4f61-8848-f82dd4835442-tls-certs\") pod \"e640ac87-e1bd-4f61-8848-f82dd4835442\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") "
Oct 30 19:47:51.534707 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.534655    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-dshm\") pod \"d3bee1cf-6278-4666-93db-a483e7dca394\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") "
Oct 30 19:47:51.534707 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.534679    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-tmp-dir\") pod \"d3bee1cf-6278-4666-93db-a483e7dca394\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") "
Oct 30 19:47:51.534882 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.534778    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-model-cache\") pod \"d3bee1cf-6278-4666-93db-a483e7dca394\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") "
Oct 30 19:47:51.534882 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.534827    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-model-cache\") pod \"e640ac87-e1bd-4f61-8848-f82dd4835442\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") "
Oct 30 19:47:51.534882 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.534879    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e640ac87-e1bd-4f61-8848-f82dd4835442-kube-api-access-274f4\" (UniqueName: \"kubernetes.io/projected/e640ac87-e1bd-4f61-8848-f82dd4835442-kube-api-access-274f4\") pod \"e640ac87-e1bd-4f61-8848-f82dd4835442\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") "
Oct 30 19:47:51.535053 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.534906    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-tmp-dir" pod "d3bee1cf-6278-4666-93db-a483e7dca394" (UID: "d3bee1cf-6278-4666-93db-a483e7dca394"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.535053 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.534934    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-tmp-dir\") pod \"e640ac87-e1bd-4f61-8848-f82dd4835442\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") "
Oct 30 19:47:51.535053 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.535015    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-home\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-home\") pod \"d3bee1cf-6278-4666-93db-a483e7dca394\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") "
Oct 30 19:47:51.535216 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.535063    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-home\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-home\") pod \"e640ac87-e1bd-4f61-8848-f82dd4835442\" (UID: \"e640ac87-e1bd-4f61-8848-f82dd4835442\") "
Oct 30 19:47:51.535216 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.535123    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3bee1cf-6278-4666-93db-a483e7dca394-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3bee1cf-6278-4666-93db-a483e7dca394-tls-certs\") pod \"d3bee1cf-6278-4666-93db-a483e7dca394\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") "
Oct 30 19:47:51.535216 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.535156    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3bee1cf-6278-4666-93db-a483e7dca394-kube-api-access-tpcxh\" (UniqueName: \"kubernetes.io/projected/d3bee1cf-6278-4666-93db-a483e7dca394-kube-api-access-tpcxh\") pod \"d3bee1cf-6278-4666-93db-a483e7dca394\" (UID: \"d3bee1cf-6278-4666-93db-a483e7dca394\") "
Oct 30 19:47:51.535393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.535056    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-model-cache" pod "d3bee1cf-6278-4666-93db-a483e7dca394" (UID: "d3bee1cf-6278-4666-93db-a483e7dca394"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.535393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.535066    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-model-cache" pod "e640ac87-e1bd-4f61-8848-f82dd4835442" (UID: "e640ac87-e1bd-4f61-8848-f82dd4835442"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.535393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.535243    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-home" pod "d3bee1cf-6278-4666-93db-a483e7dca394" (UID: "d3bee1cf-6278-4666-93db-a483e7dca394"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.535393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.535251    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-home" pod "e640ac87-e1bd-4f61-8848-f82dd4835442" (UID: "e640ac87-e1bd-4f61-8848-f82dd4835442"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.535393 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.535279    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-tmp-dir" pod "e640ac87-e1bd-4f61-8848-f82dd4835442" (UID: "e640ac87-e1bd-4f61-8848-f82dd4835442"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.535668 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.535639    4712 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-tmp-dir\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:51.535668 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.535662    4712 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-model-cache\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:51.535774 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.535676    4712 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-model-cache\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:51.535774 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.535689    4712 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-tmp-dir\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:51.535774 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.535701    4712 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-home\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:51.535774 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.535714    4712 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-home\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:51.537244 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.537206    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e640ac87-e1bd-4f61-8848-f82dd4835442-kube-api-access-274f4" pod "e640ac87-e1bd-4f61-8848-f82dd4835442" (UID: "e640ac87-e1bd-4f61-8848-f82dd4835442"). InnerVolumeSpecName "kube-api-access-274f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:51.538548 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.538527    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3bee1cf-6278-4666-93db-a483e7dca394-kube-api-access-tpcxh" pod "d3bee1cf-6278-4666-93db-a483e7dca394" (UID: "d3bee1cf-6278-4666-93db-a483e7dca394"). InnerVolumeSpecName "kube-api-access-tpcxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:51.539608 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.539586    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-dshm" pod "e640ac87-e1bd-4f61-8848-f82dd4835442" (UID: "e640ac87-e1bd-4f61-8848-f82dd4835442"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.539608 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.539600    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e640ac87-e1bd-4f61-8848-f82dd4835442-tls-certs" pod "e640ac87-e1bd-4f61-8848-f82dd4835442" (UID: "e640ac87-e1bd-4f61-8848-f82dd4835442"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:51.539729 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.539617    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-dshm" pod "d3bee1cf-6278-4666-93db-a483e7dca394" (UID: "d3bee1cf-6278-4666-93db-a483e7dca394"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:51.540756 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.540737    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3bee1cf-6278-4666-93db-a483e7dca394-tls-certs" pod "d3bee1cf-6278-4666-93db-a483e7dca394" (UID: "d3bee1cf-6278-4666-93db-a483e7dca394"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:51.636627 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.636598    4712 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3bee1cf-6278-4666-93db-a483e7dca394-tls-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:51.636627 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.636624    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpcxh\" (UniqueName: \"kubernetes.io/projected/d3bee1cf-6278-4666-93db-a483e7dca394-kube-api-access-tpcxh\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:51.636627 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.636633    4712 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e640ac87-e1bd-4f61-8848-f82dd4835442-dshm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:51.636844 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.636642    4712 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e640ac87-e1bd-4f61-8848-f82dd4835442-tls-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:51.636844 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.636651    4712 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3bee1cf-6278-4666-93db-a483e7dca394-dshm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:51.636844 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.636659    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-274f4\" (UniqueName: \"kubernetes.io/projected/e640ac87-e1bd-4f61-8848-f82dd4835442-kube-api-access-274f4\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:47:51.682216 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:51.682184    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:52.136002 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:52.135958    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c" podUID="d19408be-2ea5-4445-99a5-9094c497b95f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:47:52.482851 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:52.482814    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"
Oct 30 19:47:52.482851 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:52.482828    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8" event={"ID":"d3bee1cf-6278-4666-93db-a483e7dca394","Type":"ContainerDied","Data":"aafe1208dafa3144ef389ca71a291992b2145c2132f3d3943b8318fc0a3c109d"}
Oct 30 19:47:52.483447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:52.482885    4712 scope.go:122] "RemoveContainer" containerID="b57bb798904714d0e87194dc71c0aa595fcc5dc1f7963bfb16fef73d8614f26d"
Oct 30 19:47:52.483447 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:52.482824    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"
Oct 30 19:47:52.564973 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:52.564940    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"]
Oct 30 19:47:52.569296 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:52.569276    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/distinct-2-w1-d797d2-kserve-596db4f7dd-jvcjk"]
Oct 30 19:47:52.626555 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:52.626533    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"]
Oct 30 19:47:52.642552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:52.642523    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d797d2/premium-w1-d797d2-kserve-78c7b9b848-bmvv8"]
Oct 30 19:47:52.851446 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:52.851350    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" path="/var/lib/kubelet/pods/d3bee1cf-6278-4666-93db-a483e7dca394/volumes"
Oct 30 19:47:52.851793 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:52.851779    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" path="/var/lib/kubelet/pods/e640ac87-e1bd-4f61-8848-f82dd4835442/volumes"
Oct 30 19:47:53.272313 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:53.272282    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:54.793020 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:54.792989    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:56.112347 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:56.112315    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:56.337880 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:56.337850    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:56.499076 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:56.499023    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:56.632057 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:56.632023    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"]
Oct 30 19:47:56.632334 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:56.632274    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7" podUID="d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" containerName="istio-proxy" containerID="cri-o://e137656526b4b1397d4622cfa3721fcda8b8f39954088789b2796653f5cf1434" gracePeriod=30
Oct 30 19:47:56.757931 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:56.757858    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:57.145468 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:57.145436    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:57.330885 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:57.330857    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:58.019106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:58.019075    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:58.186828 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:58.186801    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:58.403086 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:58.403055    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:47:59.920519 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:47:59.920482    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:00.216906 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:00.216820    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:01.795135 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:01.794846    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:01.884237 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:01.884208    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:48:02.033684 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.033603    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-envoy\") pod \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") "
Oct 30 19:48:02.033684 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.033661    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-workload-socket\") pod \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") "
Oct 30 19:48:02.033922 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.033730    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-data\") pod \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") "
Oct 30 19:48:02.033922 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.033781    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-kube-api-access-z4xvb\" (UniqueName: \"kubernetes.io/projected/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-kube-api-access-z4xvb\") pod \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") "
Oct 30 19:48:02.033922 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.033820    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-workload-certs\") pod \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") "
Oct 30 19:48:02.033922 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.033868    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-token\" (UniqueName: \"kubernetes.io/projected/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-token\") pod \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") "
Oct 30 19:48:02.034124 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.033923    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-credential-socket\") pod \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") "
Oct 30 19:48:02.034124 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.033956    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-workload-socket" pod "d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" (UID: "d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:02.034124 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.033970    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-podinfo\") pod \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") "
Oct 30 19:48:02.034124 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.034061    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-data" pod "d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" (UID: "d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:02.034124 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.034074    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istiod-ca-cert\") pod \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\" (UID: \"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2\") "
Oct 30 19:48:02.034429 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.034170    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-workload-certs" pod "d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" (UID: "d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:02.034429 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.034261    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-credential-socket" pod "d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" (UID: "d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:02.034541 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.034446    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istiod-ca-cert" pod "d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" (UID: "d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:02.034541 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.034527    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:02.034648 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.034547    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:02.034648 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.034561    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:02.034648 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.034576    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:02.034648 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.034589    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:02.036001 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.035977    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-kube-api-access-z4xvb" pod "d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" (UID: "d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2"). InnerVolumeSpecName "kube-api-access-z4xvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:02.036135 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.036105    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-podinfo" pod "d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" (UID: "d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:02.036401 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.036376    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-envoy" pod "d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" (UID: "d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:02.036509 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.036431    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-token" pod "d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" (UID: "d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:02.135285 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.135259    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:02.135285 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.135288    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:02.135524 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.135299    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:02.135524 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.135309    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4xvb\" (UniqueName: \"kubernetes.io/projected/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2-kube-api-access-z4xvb\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:02.135524 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.135498    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c" podUID="d19408be-2ea5-4445-99a5-9094c497b95f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:48:02.536123 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.536093    4712 generic.go:356] "Generic (PLEG): container finished" podID="d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" containerID="e137656526b4b1397d4622cfa3721fcda8b8f39954088789b2796653f5cf1434" exitCode=0
Oct 30 19:48:02.536301 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.536138    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"
Oct 30 19:48:02.536301 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.536191    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7" event={"ID":"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2","Type":"ContainerDied","Data":"e137656526b4b1397d4622cfa3721fcda8b8f39954088789b2796653f5cf1434"}
Oct 30 19:48:02.536301 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.536262    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7" event={"ID":"d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2","Type":"ContainerDied","Data":"fb1b5dc04afdf608c4f49587a67d46df3a9daad1edc1342389a53cc1c4238cd8"}
Oct 30 19:48:02.536301 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.536294    4712 scope.go:122] "RemoveContainer" containerID="e137656526b4b1397d4622cfa3721fcda8b8f39954088789b2796653f5cf1434"
Oct 30 19:48:02.553761 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.553734    4712 scope.go:122] "RemoveContainer" containerID="e137656526b4b1397d4622cfa3721fcda8b8f39954088789b2796653f5cf1434"
Oct 30 19:48:02.554111 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:48:02.554079    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e137656526b4b1397d4622cfa3721fcda8b8f39954088789b2796653f5cf1434\": container with ID starting with e137656526b4b1397d4622cfa3721fcda8b8f39954088789b2796653f5cf1434 not found: ID does not exist" containerID="e137656526b4b1397d4622cfa3721fcda8b8f39954088789b2796653f5cf1434"
Oct 30 19:48:02.554197 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.554117    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e137656526b4b1397d4622cfa3721fcda8b8f39954088789b2796653f5cf1434"} err="failed to get container status \"e137656526b4b1397d4622cfa3721fcda8b8f39954088789b2796653f5cf1434\": rpc error: code = NotFound desc = could not find container \"e137656526b4b1397d4622cfa3721fcda8b8f39954088789b2796653f5cf1434\": container with ID starting with e137656526b4b1397d4622cfa3721fcda8b8f39954088789b2796653f5cf1434 not found: ID does not exist"
Oct 30 19:48:02.594610 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.594584    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"]
Oct 30 19:48:02.606378 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.606341    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f913c6-openshift-default-85664fdcfc-m48x7"]
Oct 30 19:48:02.851996 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:02.851927    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" path="/var/lib/kubelet/pods/d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2/volumes"
Oct 30 19:48:04.068891 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:04.068858    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:05.563450 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:05.563412    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:05.997861 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:05.997831    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:06.218675 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:06.218633    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:06.372614 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:48:06.372582    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:48:06.376881 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:06.376862    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:06.657316 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:06.657250    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:07.210326 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:07.210270    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c"
Oct 30 19:48:07.295945 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:07.295914    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d19408be-2ea5-4445-99a5-9094c497b95f-kube-api-access-49hjj\" (UniqueName: \"kubernetes.io/projected/d19408be-2ea5-4445-99a5-9094c497b95f-kube-api-access-49hjj\") pod \"d19408be-2ea5-4445-99a5-9094c497b95f\" (UID: \"d19408be-2ea5-4445-99a5-9094c497b95f\") "
Oct 30 19:48:07.296087 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:07.296033    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d19408be-2ea5-4445-99a5-9094c497b95f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19408be-2ea5-4445-99a5-9094c497b95f-plugins-config-volume\") pod \"d19408be-2ea5-4445-99a5-9094c497b95f\" (UID: \"d19408be-2ea5-4445-99a5-9094c497b95f\") "
Oct 30 19:48:07.296445 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:07.296414    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d19408be-2ea5-4445-99a5-9094c497b95f-plugins-config-volume" pod "d19408be-2ea5-4445-99a5-9094c497b95f" (UID: "d19408be-2ea5-4445-99a5-9094c497b95f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:07.298255 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:07.298225    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19408be-2ea5-4445-99a5-9094c497b95f-kube-api-access-49hjj" pod "d19408be-2ea5-4445-99a5-9094c497b95f" (UID: "d19408be-2ea5-4445-99a5-9094c497b95f"). InnerVolumeSpecName "kube-api-access-49hjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:07.396926 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:07.396903    4712 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19408be-2ea5-4445-99a5-9094c497b95f-plugins-config-volume\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:07.396926 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:07.396924    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49hjj\" (UniqueName: \"kubernetes.io/projected/d19408be-2ea5-4445-99a5-9094c497b95f-kube-api-access-49hjj\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:07.563527 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:07.563455    4712 generic.go:356] "Generic (PLEG): container finished" podID="d19408be-2ea5-4445-99a5-9094c497b95f" containerID="6122c058157a02ff805345e35e90d1c10ab70a725c5d40017f7d6d400815d43d" exitCode=0
Oct 30 19:48:07.563527 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:07.563504    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c"
Oct 30 19:48:07.563527 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:07.563524    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c" event={"ID":"d19408be-2ea5-4445-99a5-9094c497b95f","Type":"ContainerDied","Data":"6122c058157a02ff805345e35e90d1c10ab70a725c5d40017f7d6d400815d43d"}
Oct 30 19:48:07.563723 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:07.563572    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c" event={"ID":"d19408be-2ea5-4445-99a5-9094c497b95f","Type":"ContainerDied","Data":"2dda31542479d740830f375741e5fa77b27cc061b613be7a2772bcc1cbb2d49c"}
Oct 30 19:48:07.563723 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:07.563603    4712 scope.go:122] "RemoveContainer" containerID="6122c058157a02ff805345e35e90d1c10ab70a725c5d40017f7d6d400815d43d"
Oct 30 19:48:07.579018 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:07.578987    4712 scope.go:122] "RemoveContainer" containerID="6122c058157a02ff805345e35e90d1c10ab70a725c5d40017f7d6d400815d43d"
Oct 30 19:48:07.579665 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:48:07.579634    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6122c058157a02ff805345e35e90d1c10ab70a725c5d40017f7d6d400815d43d\": container with ID starting with 6122c058157a02ff805345e35e90d1c10ab70a725c5d40017f7d6d400815d43d not found: ID does not exist" containerID="6122c058157a02ff805345e35e90d1c10ab70a725c5d40017f7d6d400815d43d"
Oct 30 19:48:07.579784 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:07.579672    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6122c058157a02ff805345e35e90d1c10ab70a725c5d40017f7d6d400815d43d"} err="failed to get container status \"6122c058157a02ff805345e35e90d1c10ab70a725c5d40017f7d6d400815d43d\": rpc error: code = NotFound desc = could not find container \"6122c058157a02ff805345e35e90d1c10ab70a725c5d40017f7d6d400815d43d\": container with ID starting with 6122c058157a02ff805345e35e90d1c10ab70a725c5d40017f7d6d400815d43d not found: ID does not exist"
Oct 30 19:48:07.614765 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:07.614739    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c"]
Oct 30 19:48:07.620297 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:07.620272    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d797d2-756b4c4bb-hvh4c"]
Oct 30 19:48:07.620862 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:07.620842    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:07.800339 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:07.800312    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:08.087864 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:08.087832    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:08.237859 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:08.237825    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:08.851377 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:08.851334    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d19408be-2ea5-4445-99a5-9094c497b95f" path="/var/lib/kubelet/pods/d19408be-2ea5-4445-99a5-9094c497b95f/volumes"
Oct 30 19:48:09.396577 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:09.396516    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:11.104479 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:11.104444    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:11.284013 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:11.283979    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:13.171472 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:13.171428    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:13.426925 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:13.426851    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:13.579571 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:13.579537    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:14.705374 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:14.705329    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:15.709911 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:15.709879    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:15.910863 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:15.910826    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:15.925127 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:15.925100    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"]
Oct 30 19:48:15.925298 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:15.925279    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh" podUID="bb85c12b-5f17-4e75-9a68-96e9f83389b1" containerName="istio-proxy" containerID="cri-o://6f7ca7d42ef85b6955f6d058d4e7552c2677e695b0b3d74b5f863db8a962b564" gracePeriod=30
Oct 30 19:48:16.123682 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:16.123647    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:16.415254 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:16.415169    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:17.182947 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:17.182905    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:18.086826 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:18.086777    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:19.539934 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:19.539902    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:21.093675 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.093642    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:21.207504 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.207475    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:48:21.223356 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.223327    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istiod-ca-cert\") pod \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") "
Oct 30 19:48:21.223543 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.223403    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-podinfo\") pod \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") "
Oct 30 19:48:21.223543 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.223439    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-data\") pod \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") "
Oct 30 19:48:21.223543 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.223471    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-credential-socket\") pod \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") "
Oct 30 19:48:21.223768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.223617    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb85c12b-5f17-4e75-9a68-96e9f83389b1-kube-api-access-8fxq8\" (UniqueName: \"kubernetes.io/projected/bb85c12b-5f17-4e75-9a68-96e9f83389b1-kube-api-access-8fxq8\") pod \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") "
Oct 30 19:48:21.223768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.223650    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-token\" (UniqueName: \"kubernetes.io/projected/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-token\") pod \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") "
Oct 30 19:48:21.223768 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.223706    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-envoy\") pod \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") "
Oct 30 19:48:21.223925 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.223770    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-workload-certs\") pod \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") "
Oct 30 19:48:21.223925 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.223771    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istiod-ca-cert" pod "bb85c12b-5f17-4e75-9a68-96e9f83389b1" (UID: "bb85c12b-5f17-4e75-9a68-96e9f83389b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:21.223925 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.223829    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-workload-socket\") pod \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\" (UID: \"bb85c12b-5f17-4e75-9a68-96e9f83389b1\") "
Oct 30 19:48:21.224321 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.224101    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-data" pod "bb85c12b-5f17-4e75-9a68-96e9f83389b1" (UID: "bb85c12b-5f17-4e75-9a68-96e9f83389b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:21.224321 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.224215    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:21.224321 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.224231    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:21.224600 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.224382    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-workload-socket" pod "bb85c12b-5f17-4e75-9a68-96e9f83389b1" (UID: "bb85c12b-5f17-4e75-9a68-96e9f83389b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:21.224600 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.224398    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-credential-socket" pod "bb85c12b-5f17-4e75-9a68-96e9f83389b1" (UID: "bb85c12b-5f17-4e75-9a68-96e9f83389b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:21.224926 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.224899    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-workload-certs" pod "bb85c12b-5f17-4e75-9a68-96e9f83389b1" (UID: "bb85c12b-5f17-4e75-9a68-96e9f83389b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:21.226502 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.226471    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-envoy" pod "bb85c12b-5f17-4e75-9a68-96e9f83389b1" (UID: "bb85c12b-5f17-4e75-9a68-96e9f83389b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:21.226753 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.226715    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-podinfo" pod "bb85c12b-5f17-4e75-9a68-96e9f83389b1" (UID: "bb85c12b-5f17-4e75-9a68-96e9f83389b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:21.226986 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.226964    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb85c12b-5f17-4e75-9a68-96e9f83389b1-kube-api-access-8fxq8" pod "bb85c12b-5f17-4e75-9a68-96e9f83389b1" (UID: "bb85c12b-5f17-4e75-9a68-96e9f83389b1"). InnerVolumeSpecName "kube-api-access-8fxq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:21.229905 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.229875    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-token" pod "bb85c12b-5f17-4e75-9a68-96e9f83389b1" (UID: "bb85c12b-5f17-4e75-9a68-96e9f83389b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:21.254695 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.254671    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:21.325497 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.325467    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:21.325622 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.325501    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:21.325622 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.325517    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:21.325622 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.325531    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:21.325622 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.325545    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fxq8\" (UniqueName: \"kubernetes.io/projected/bb85c12b-5f17-4e75-9a68-96e9f83389b1-kube-api-access-8fxq8\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:21.325622 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.325559    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:21.325622 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.325571    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb85c12b-5f17-4e75-9a68-96e9f83389b1-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:21.635410 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.635379    4712 generic.go:356] "Generic (PLEG): container finished" podID="bb85c12b-5f17-4e75-9a68-96e9f83389b1" containerID="6f7ca7d42ef85b6955f6d058d4e7552c2677e695b0b3d74b5f863db8a962b564" exitCode=0
Oct 30 19:48:21.635609 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.635425    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"
Oct 30 19:48:21.635609 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.635460    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh" event={"ID":"bb85c12b-5f17-4e75-9a68-96e9f83389b1","Type":"ContainerDied","Data":"6f7ca7d42ef85b6955f6d058d4e7552c2677e695b0b3d74b5f863db8a962b564"}
Oct 30 19:48:21.635609 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.635524    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh" event={"ID":"bb85c12b-5f17-4e75-9a68-96e9f83389b1","Type":"ContainerDied","Data":"52276e4974d8ae155c9d3a54da838e62660cc10793492bce5ce2f6381205c820"}
Oct 30 19:48:21.635609 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.635556    4712 scope.go:122] "RemoveContainer" containerID="6f7ca7d42ef85b6955f6d058d4e7552c2677e695b0b3d74b5f863db8a962b564"
Oct 30 19:48:21.645155 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.645137    4712 scope.go:122] "RemoveContainer" containerID="6f7ca7d42ef85b6955f6d058d4e7552c2677e695b0b3d74b5f863db8a962b564"
Oct 30 19:48:21.645434 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:48:21.645418    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f7ca7d42ef85b6955f6d058d4e7552c2677e695b0b3d74b5f863db8a962b564\": container with ID starting with 6f7ca7d42ef85b6955f6d058d4e7552c2677e695b0b3d74b5f863db8a962b564 not found: ID does not exist" containerID="6f7ca7d42ef85b6955f6d058d4e7552c2677e695b0b3d74b5f863db8a962b564"
Oct 30 19:48:21.645488 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.645442    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f7ca7d42ef85b6955f6d058d4e7552c2677e695b0b3d74b5f863db8a962b564"} err="failed to get container status \"6f7ca7d42ef85b6955f6d058d4e7552c2677e695b0b3d74b5f863db8a962b564\": rpc error: code = NotFound desc = could not find container \"6f7ca7d42ef85b6955f6d058d4e7552c2677e695b0b3d74b5f863db8a962b564\": container with ID starting with 6f7ca7d42ef85b6955f6d058d4e7552c2677e695b0b3d74b5f863db8a962b564 not found: ID does not exist"
Oct 30 19:48:21.686795 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.686773    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"]
Oct 30 19:48:21.698661 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:21.698641    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d797d2-openshift-default-6dbbffdb8f-v57rh"]
Oct 30 19:48:22.496516 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:22.496476    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:22.851516 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:22.851430    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb85c12b-5f17-4e75-9a68-96e9f83389b1" path="/var/lib/kubelet/pods/bb85c12b-5f17-4e75-9a68-96e9f83389b1/volumes"
Oct 30 19:48:23.187354 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:23.187323    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:23.884498 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:23.884469    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:24.974576 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:24.974542    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"]
Oct 30 19:48:24.975075 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:24.974739    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv" podUID="60db5e44-82df-473c-9fc5-f7682f2c7000" containerName="maas-api" containerID="cri-o://be5e552f33a191949cd6f7405b13bb507aa4c4e490d0a894ec09d164c482f315" gracePeriod=30
Oct 30 19:48:25.097785 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.097749    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:25.228623 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.228549    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"
Oct 30 19:48:25.264385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.264331    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60db5e44-82df-473c-9fc5-f7682f2c7000-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60db5e44-82df-473c-9fc5-f7682f2c7000-openshift-service-ca\") pod \"60db5e44-82df-473c-9fc5-f7682f2c7000\" (UID: \"60db5e44-82df-473c-9fc5-f7682f2c7000\") "
Oct 30 19:48:25.264545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.264473    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60db5e44-82df-473c-9fc5-f7682f2c7000-kube-api-access-7kbtn\" (UniqueName: \"kubernetes.io/projected/60db5e44-82df-473c-9fc5-f7682f2c7000-kube-api-access-7kbtn\") pod \"60db5e44-82df-473c-9fc5-f7682f2c7000\" (UID: \"60db5e44-82df-473c-9fc5-f7682f2c7000\") "
Oct 30 19:48:25.264592 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.264560    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60db5e44-82df-473c-9fc5-f7682f2c7000-metrics-certs\" (UniqueName: \"kubernetes.io/secret/60db5e44-82df-473c-9fc5-f7682f2c7000-metrics-certs\") pod \"60db5e44-82df-473c-9fc5-f7682f2c7000\" (UID: \"60db5e44-82df-473c-9fc5-f7682f2c7000\") "
Oct 30 19:48:25.264636 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.264593    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60db5e44-82df-473c-9fc5-f7682f2c7000-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60db5e44-82df-473c-9fc5-f7682f2c7000-maas-api-tls\") pod \"60db5e44-82df-473c-9fc5-f7682f2c7000\" (UID: \"60db5e44-82df-473c-9fc5-f7682f2c7000\") "
Oct 30 19:48:25.264770 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.264750    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60db5e44-82df-473c-9fc5-f7682f2c7000-openshift-service-ca" pod "60db5e44-82df-473c-9fc5-f7682f2c7000" (UID: "60db5e44-82df-473c-9fc5-f7682f2c7000"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:25.264975 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.264956    4712 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60db5e44-82df-473c-9fc5-f7682f2c7000-openshift-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:25.266924 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.266890    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60db5e44-82df-473c-9fc5-f7682f2c7000-kube-api-access-7kbtn" pod "60db5e44-82df-473c-9fc5-f7682f2c7000" (UID: "60db5e44-82df-473c-9fc5-f7682f2c7000"). InnerVolumeSpecName "kube-api-access-7kbtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:25.267048 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.266924    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60db5e44-82df-473c-9fc5-f7682f2c7000-metrics-certs" pod "60db5e44-82df-473c-9fc5-f7682f2c7000" (UID: "60db5e44-82df-473c-9fc5-f7682f2c7000"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:25.267209 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.267184    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60db5e44-82df-473c-9fc5-f7682f2c7000-maas-api-tls" pod "60db5e44-82df-473c-9fc5-f7682f2c7000" (UID: "60db5e44-82df-473c-9fc5-f7682f2c7000"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:25.366306 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.366277    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kbtn\" (UniqueName: \"kubernetes.io/projected/60db5e44-82df-473c-9fc5-f7682f2c7000-kube-api-access-7kbtn\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:25.366306 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.366304    4712 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60db5e44-82df-473c-9fc5-f7682f2c7000-metrics-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:25.366306 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.366313    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60db5e44-82df-473c-9fc5-f7682f2c7000-maas-api-tls\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:48:25.595462 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.595325    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:25.658001 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.657970    4712 generic.go:356] "Generic (PLEG): container finished" podID="60db5e44-82df-473c-9fc5-f7682f2c7000" containerID="be5e552f33a191949cd6f7405b13bb507aa4c4e490d0a894ec09d164c482f315" exitCode=0
Oct 30 19:48:25.658151 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.658017    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"
Oct 30 19:48:25.658151 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.658010    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv" event={"ID":"60db5e44-82df-473c-9fc5-f7682f2c7000","Type":"ContainerDied","Data":"be5e552f33a191949cd6f7405b13bb507aa4c4e490d0a894ec09d164c482f315"}
Oct 30 19:48:25.658151 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.658142    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv" event={"ID":"60db5e44-82df-473c-9fc5-f7682f2c7000","Type":"ContainerDied","Data":"bcfb523a803abf066fa95034ae76338d825f4dfd257fdac5d5ac411b8358a87e"}
Oct 30 19:48:25.658295 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.658172    4712 scope.go:122] "RemoveContainer" containerID="be5e552f33a191949cd6f7405b13bb507aa4c4e490d0a894ec09d164c482f315"
Oct 30 19:48:25.667547 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.667522    4712 scope.go:122] "RemoveContainer" containerID="be5e552f33a191949cd6f7405b13bb507aa4c4e490d0a894ec09d164c482f315"
Oct 30 19:48:25.667818 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:48:25.667798    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5e552f33a191949cd6f7405b13bb507aa4c4e490d0a894ec09d164c482f315\": container with ID starting with be5e552f33a191949cd6f7405b13bb507aa4c4e490d0a894ec09d164c482f315 not found: ID does not exist" containerID="be5e552f33a191949cd6f7405b13bb507aa4c4e490d0a894ec09d164c482f315"
Oct 30 19:48:25.667882 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.667826    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5e552f33a191949cd6f7405b13bb507aa4c4e490d0a894ec09d164c482f315"} err="failed to get container status \"be5e552f33a191949cd6f7405b13bb507aa4c4e490d0a894ec09d164c482f315\": rpc error: code = NotFound desc = could not find container \"be5e552f33a191949cd6f7405b13bb507aa4c4e490d0a894ec09d164c482f315\": container with ID starting with be5e552f33a191949cd6f7405b13bb507aa4c4e490d0a894ec09d164c482f315 not found: ID does not exist"
Oct 30 19:48:25.711708 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.711671    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"]
Oct 30 19:48:25.714928 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.714887    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-3c89af-7c49956c-q28rv"]
Oct 30 19:48:25.746263 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:25.746238    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:26.246946 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:26.246915    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:26.850539 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:26.850508    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60db5e44-82df-473c-9fc5-f7682f2c7000" path="/var/lib/kubelet/pods/60db5e44-82df-473c-9fc5-f7682f2c7000/volumes"
Oct 30 19:48:27.384219 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:27.384189    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:28.917147 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:28.917062    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:30.363563 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:30.363523    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:31.897933 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:31.897884    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:32.122940 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:32.122907    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:32.313575 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:32.313491    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:32.495829 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:32.495776    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:33.444513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:33.444477    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:33.598341 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:33.598304    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:35.156460 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:35.156427    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:36.373438 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:48:36.373406    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:48:36.669717 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:36.669635    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:38.198573 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:38.198521    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:39.694106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:39.694069    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:41.109552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:41.109439    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:42.590160 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:42.590114    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:43.539104 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:43.539075    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:43.739125 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:43.739088    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:43.997404 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:43.997356    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:45.185179 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:45.185152    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:45.636560 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:45.636530    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:45.782421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:45.782387    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:46.271149 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:46.271116    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:47.327467 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:47.327432    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:47.935859 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:47.935831    4712 scope.go:122] "RemoveContainer" containerID="5bf841fa8a53fbc255a62c00e33fedd2bc42b7e233c7757eba6b88a0777d61c9"
Oct 30 19:48:49.010703 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:49.010670    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:50.475164 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:50.475130    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:52.002048 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:52.002010    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:53.188538 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:53.188504    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:53.409990 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:53.409956    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:53.581385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:53.581292    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:53.851070 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:53.850979    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:54.024429 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:54.024387    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:54.165751 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:54.165721    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:55.510879 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:55.510849    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:56.985321 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:56.985284    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:58.624093 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:58.624051    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:48:58.821257 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:58.821224    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"]
Oct 30 19:48:58.850826 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:58.850799    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"]
Oct 30 19:48:59.122590 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:48:59.122558    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:00.372106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:00.372070    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:01.792511 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:01.792467    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:04.360450 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:04.360416    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:04.697076 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:04.697043    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:04.897605 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:04.897569    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:06.373725 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:49:06.373689    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:49:13.841288 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:13.841225    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" containerID="cri-o://350b4d281e03a0648a9702028d22d689f3fcd16b4f332e4d6eb15c207f9f61de" gracePeriod=45
Oct 30 19:49:13.869403 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:13.869344    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main" containerID="cri-o://3713b43ac06964f79842389acdb50f7b1938cc86ac407a27212bcaac43fbceeb" gracePeriod=45
Oct 30 19:49:14.016036 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.015982    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": dial tcp 10.133.0.70:8000: connect: connection refused"
Oct 30 19:49:14.113697 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.113668    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:49:14.143900 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.143878    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:49:14.215213 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.215179    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-dshm\") pod \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") "
Oct 30 19:49:14.215391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.215220    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a6dccf3-9a50-43ef-833f-1e66fe53890f-kube-api-access-h95hb\" (UniqueName: \"kubernetes.io/projected/0a6dccf3-9a50-43ef-833f-1e66fe53890f-kube-api-access-h95hb\") pod \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") "
Oct 30 19:49:14.215391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.215239    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-dshm\") pod \"946e2596-1023-4d05-ba84-2dd3da74cc8e\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") "
Oct 30 19:49:14.215391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.215270    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/946e2596-1023-4d05-ba84-2dd3da74cc8e-tls-certs\" (UniqueName: \"kubernetes.io/secret/946e2596-1023-4d05-ba84-2dd3da74cc8e-tls-certs\") pod \"946e2596-1023-4d05-ba84-2dd3da74cc8e\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") "
Oct 30 19:49:14.215391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.215327    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946e2596-1023-4d05-ba84-2dd3da74cc8e-kube-api-access-slrzx\" (UniqueName: \"kubernetes.io/projected/946e2596-1023-4d05-ba84-2dd3da74cc8e-kube-api-access-slrzx\") pod \"946e2596-1023-4d05-ba84-2dd3da74cc8e\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") "
Oct 30 19:49:14.215391 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.215357    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-tmp-dir\") pod \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") "
Oct 30 19:49:14.215684 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.215418    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-tmp-dir\") pod \"946e2596-1023-4d05-ba84-2dd3da74cc8e\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") "
Oct 30 19:49:14.215684 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.215453    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a6dccf3-9a50-43ef-833f-1e66fe53890f-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a6dccf3-9a50-43ef-833f-1e66fe53890f-tls-certs\") pod \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") "
Oct 30 19:49:14.215684 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.215495    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-model-cache\") pod \"946e2596-1023-4d05-ba84-2dd3da74cc8e\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") "
Oct 30 19:49:14.215684 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.215540    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-home\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-home\") pod \"946e2596-1023-4d05-ba84-2dd3da74cc8e\" (UID: \"946e2596-1023-4d05-ba84-2dd3da74cc8e\") "
Oct 30 19:49:14.215684 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.215571    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-home\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-home\") pod \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") "
Oct 30 19:49:14.215684 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.215662    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-model-cache\") pod \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\" (UID: \"0a6dccf3-9a50-43ef-833f-1e66fe53890f\") "
Oct 30 19:49:14.216002 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.215879    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-tmp-dir" pod "946e2596-1023-4d05-ba84-2dd3da74cc8e" (UID: "946e2596-1023-4d05-ba84-2dd3da74cc8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:14.216138 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.216109    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-model-cache" pod "0a6dccf3-9a50-43ef-833f-1e66fe53890f" (UID: "0a6dccf3-9a50-43ef-833f-1e66fe53890f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:14.216396 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.216312    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-model-cache" pod "946e2596-1023-4d05-ba84-2dd3da74cc8e" (UID: "946e2596-1023-4d05-ba84-2dd3da74cc8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:14.216396 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.216132    4712 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-tmp-dir\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:14.216396 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.216398    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-home" pod "946e2596-1023-4d05-ba84-2dd3da74cc8e" (UID: "946e2596-1023-4d05-ba84-2dd3da74cc8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:14.216648 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.216450    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-tmp-dir" pod "0a6dccf3-9a50-43ef-833f-1e66fe53890f" (UID: "0a6dccf3-9a50-43ef-833f-1e66fe53890f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:14.216648 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.216489    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-home" pod "0a6dccf3-9a50-43ef-833f-1e66fe53890f" (UID: "0a6dccf3-9a50-43ef-833f-1e66fe53890f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:14.218402 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.218376    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a6dccf3-9a50-43ef-833f-1e66fe53890f-kube-api-access-h95hb" pod "0a6dccf3-9a50-43ef-833f-1e66fe53890f" (UID: "0a6dccf3-9a50-43ef-833f-1e66fe53890f"). InnerVolumeSpecName "kube-api-access-h95hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:14.218539 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.218516    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-dshm" pod "0a6dccf3-9a50-43ef-833f-1e66fe53890f" (UID: "0a6dccf3-9a50-43ef-833f-1e66fe53890f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:14.218638 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.218613    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946e2596-1023-4d05-ba84-2dd3da74cc8e-kube-api-access-slrzx" pod "946e2596-1023-4d05-ba84-2dd3da74cc8e" (UID: "946e2596-1023-4d05-ba84-2dd3da74cc8e"). InnerVolumeSpecName "kube-api-access-slrzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:14.218717 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.218651    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/946e2596-1023-4d05-ba84-2dd3da74cc8e-tls-certs" pod "946e2596-1023-4d05-ba84-2dd3da74cc8e" (UID: "946e2596-1023-4d05-ba84-2dd3da74cc8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:14.218762 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.218738    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-dshm" pod "946e2596-1023-4d05-ba84-2dd3da74cc8e" (UID: "946e2596-1023-4d05-ba84-2dd3da74cc8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:14.218894 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.218873    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a6dccf3-9a50-43ef-833f-1e66fe53890f-tls-certs" pod "0a6dccf3-9a50-43ef-833f-1e66fe53890f" (UID: "0a6dccf3-9a50-43ef-833f-1e66fe53890f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:14.317844 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.317808    4712 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-dshm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:14.317844 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.317840    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h95hb\" (UniqueName: \"kubernetes.io/projected/0a6dccf3-9a50-43ef-833f-1e66fe53890f-kube-api-access-h95hb\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:14.317844 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.317851    4712 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-dshm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:14.318070 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.317859    4712 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/946e2596-1023-4d05-ba84-2dd3da74cc8e-tls-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:14.318070 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.317870    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slrzx\" (UniqueName: \"kubernetes.io/projected/946e2596-1023-4d05-ba84-2dd3da74cc8e-kube-api-access-slrzx\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:14.318070 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.317878    4712 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-tmp-dir\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:14.318070 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.317886    4712 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a6dccf3-9a50-43ef-833f-1e66fe53890f-tls-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:14.318070 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.317894    4712 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-model-cache\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:14.318070 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.317901    4712 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/946e2596-1023-4d05-ba84-2dd3da74cc8e-home\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:14.318070 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.317908    4712 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-home\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:14.318070 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.317915    4712 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a6dccf3-9a50-43ef-833f-1e66fe53890f-model-cache\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:14.897832 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.897801    4712 generic.go:356] "Generic (PLEG): container finished" podID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerID="350b4d281e03a0648a9702028d22d689f3fcd16b4f332e4d6eb15c207f9f61de" exitCode=0
Oct 30 19:49:14.898187 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.897835    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"
Oct 30 19:49:14.898187 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.897874    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" event={"ID":"946e2596-1023-4d05-ba84-2dd3da74cc8e","Type":"ContainerDied","Data":"350b4d281e03a0648a9702028d22d689f3fcd16b4f332e4d6eb15c207f9f61de"}
Oct 30 19:49:14.898187 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.897909    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl" event={"ID":"946e2596-1023-4d05-ba84-2dd3da74cc8e","Type":"ContainerDied","Data":"ad017e08bd3bb3069aed4482b2b2ab8fa43948cd512e32a9e8cbaae66e19a0e3"}
Oct 30 19:49:14.898187 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.897925    4712 scope.go:122] "RemoveContainer" containerID="350b4d281e03a0648a9702028d22d689f3fcd16b4f332e4d6eb15c207f9f61de"
Oct 30 19:49:14.899357 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.899291    4712 generic.go:356] "Generic (PLEG): container finished" podID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerID="3713b43ac06964f79842389acdb50f7b1938cc86ac407a27212bcaac43fbceeb" exitCode=0
Oct 30 19:49:14.899357 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.899328    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"
Oct 30 19:49:14.899514 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.899380    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" event={"ID":"0a6dccf3-9a50-43ef-833f-1e66fe53890f","Type":"ContainerDied","Data":"3713b43ac06964f79842389acdb50f7b1938cc86ac407a27212bcaac43fbceeb"}
Oct 30 19:49:14.899514 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.899403    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8" event={"ID":"0a6dccf3-9a50-43ef-833f-1e66fe53890f","Type":"ContainerDied","Data":"d660850c639aace98a2307bfe9973782dc3066a21edd2ae441da2550d37a19cd"}
Oct 30 19:49:14.906466 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.906440    4712 scope.go:122] "RemoveContainer" containerID="350b4d281e03a0648a9702028d22d689f3fcd16b4f332e4d6eb15c207f9f61de"
Oct 30 19:49:14.906726 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:49:14.906705    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350b4d281e03a0648a9702028d22d689f3fcd16b4f332e4d6eb15c207f9f61de\": container with ID starting with 350b4d281e03a0648a9702028d22d689f3fcd16b4f332e4d6eb15c207f9f61de not found: ID does not exist" containerID="350b4d281e03a0648a9702028d22d689f3fcd16b4f332e4d6eb15c207f9f61de"
Oct 30 19:49:14.906805 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.906738    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350b4d281e03a0648a9702028d22d689f3fcd16b4f332e4d6eb15c207f9f61de"} err="failed to get container status \"350b4d281e03a0648a9702028d22d689f3fcd16b4f332e4d6eb15c207f9f61de\": rpc error: code = NotFound desc = could not find container \"350b4d281e03a0648a9702028d22d689f3fcd16b4f332e4d6eb15c207f9f61de\": container with ID starting with 350b4d281e03a0648a9702028d22d689f3fcd16b4f332e4d6eb15c207f9f61de not found: ID does not exist"
Oct 30 19:49:14.906805 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.906762    4712 scope.go:122] "RemoveContainer" containerID="3713b43ac06964f79842389acdb50f7b1938cc86ac407a27212bcaac43fbceeb"
Oct 30 19:49:14.913835 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.913818    4712 scope.go:122] "RemoveContainer" containerID="3713b43ac06964f79842389acdb50f7b1938cc86ac407a27212bcaac43fbceeb"
Oct 30 19:49:14.914088 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:49:14.914069    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3713b43ac06964f79842389acdb50f7b1938cc86ac407a27212bcaac43fbceeb\": container with ID starting with 3713b43ac06964f79842389acdb50f7b1938cc86ac407a27212bcaac43fbceeb not found: ID does not exist" containerID="3713b43ac06964f79842389acdb50f7b1938cc86ac407a27212bcaac43fbceeb"
Oct 30 19:49:14.914135 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.914094    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3713b43ac06964f79842389acdb50f7b1938cc86ac407a27212bcaac43fbceeb"} err="failed to get container status \"3713b43ac06964f79842389acdb50f7b1938cc86ac407a27212bcaac43fbceeb\": rpc error: code = NotFound desc = could not find container \"3713b43ac06964f79842389acdb50f7b1938cc86ac407a27212bcaac43fbceeb\": container with ID starting with 3713b43ac06964f79842389acdb50f7b1938cc86ac407a27212bcaac43fbceeb not found: ID does not exist"
Oct 30 19:49:14.951271 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.951247    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"]
Oct 30 19:49:14.954028 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.954010    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/premium-w5-544fc8-kserve-554c74c5d9-2gmk8"]
Oct 30 19:49:14.997179 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.997156    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"]
Oct 30 19:49:14.999981 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:14.999961    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-544fc8/distinct-2-w5-544fc8-kserve-7cc458ccf9-x8hvl"]
Oct 30 19:49:16.853726 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:16.853690    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" path="/var/lib/kubelet/pods/0a6dccf3-9a50-43ef-833f-1e66fe53890f/volumes"
Oct 30 19:49:16.854314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:16.854291    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" path="/var/lib/kubelet/pods/946e2596-1023-4d05-ba84-2dd3da74cc8e/volumes"
Oct 30 19:49:21.007910 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.007880    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"]
Oct 30 19:49:21.008906 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.008085    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2" podUID="aa47487f-ee16-4984-9091-1f277d564af2" containerName="maas-api" containerID="cri-o://d416c951ebbb5acb038fe7f0f5eb253a3404a3a81206c77514b9c8b8fd56f300" gracePeriod=30
Oct 30 19:49:21.128539 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.128504    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:21.258077 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.258003    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"
Oct 30 19:49:21.381083 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.381052    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa47487f-ee16-4984-9091-1f277d564af2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa47487f-ee16-4984-9091-1f277d564af2-maas-api-tls\") pod \"aa47487f-ee16-4984-9091-1f277d564af2\" (UID: \"aa47487f-ee16-4984-9091-1f277d564af2\") "
Oct 30 19:49:21.381245 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.381136    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa47487f-ee16-4984-9091-1f277d564af2-kube-api-access-dcbbj\" (UniqueName: \"kubernetes.io/projected/aa47487f-ee16-4984-9091-1f277d564af2-kube-api-access-dcbbj\") pod \"aa47487f-ee16-4984-9091-1f277d564af2\" (UID: \"aa47487f-ee16-4984-9091-1f277d564af2\") "
Oct 30 19:49:21.381245 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.381174    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa47487f-ee16-4984-9091-1f277d564af2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa47487f-ee16-4984-9091-1f277d564af2-openshift-service-ca\") pod \"aa47487f-ee16-4984-9091-1f277d564af2\" (UID: \"aa47487f-ee16-4984-9091-1f277d564af2\") "
Oct 30 19:49:21.381245 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.381207    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa47487f-ee16-4984-9091-1f277d564af2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa47487f-ee16-4984-9091-1f277d564af2-metrics-certs\") pod \"aa47487f-ee16-4984-9091-1f277d564af2\" (UID: \"aa47487f-ee16-4984-9091-1f277d564af2\") "
Oct 30 19:49:21.381601 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.381576    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa47487f-ee16-4984-9091-1f277d564af2-openshift-service-ca" pod "aa47487f-ee16-4984-9091-1f277d564af2" (UID: "aa47487f-ee16-4984-9091-1f277d564af2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:21.383154 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.383132    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa47487f-ee16-4984-9091-1f277d564af2-maas-api-tls" pod "aa47487f-ee16-4984-9091-1f277d564af2" (UID: "aa47487f-ee16-4984-9091-1f277d564af2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:21.383293 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.383275    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa47487f-ee16-4984-9091-1f277d564af2-metrics-certs" pod "aa47487f-ee16-4984-9091-1f277d564af2" (UID: "aa47487f-ee16-4984-9091-1f277d564af2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:21.383425 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.383403    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa47487f-ee16-4984-9091-1f277d564af2-kube-api-access-dcbbj" pod "aa47487f-ee16-4984-9091-1f277d564af2" (UID: "aa47487f-ee16-4984-9091-1f277d564af2"). InnerVolumeSpecName "kube-api-access-dcbbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:21.482697 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.482666    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcbbj\" (UniqueName: \"kubernetes.io/projected/aa47487f-ee16-4984-9091-1f277d564af2-kube-api-access-dcbbj\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:21.482697 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.482695    4712 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa47487f-ee16-4984-9091-1f277d564af2-openshift-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:21.482888 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.482706    4712 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa47487f-ee16-4984-9091-1f277d564af2-metrics-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:21.482888 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.482715    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa47487f-ee16-4984-9091-1f277d564af2-maas-api-tls\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:21.550950 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.550883    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:21.730164 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.730131    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:21.934971 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.934938    4712 generic.go:356] "Generic (PLEG): container finished" podID="aa47487f-ee16-4984-9091-1f277d564af2" containerID="d416c951ebbb5acb038fe7f0f5eb253a3404a3a81206c77514b9c8b8fd56f300" exitCode=0
Oct 30 19:49:21.935103 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.934997    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"
Oct 30 19:49:21.935103 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.935021    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2" event={"ID":"aa47487f-ee16-4984-9091-1f277d564af2","Type":"ContainerDied","Data":"d416c951ebbb5acb038fe7f0f5eb253a3404a3a81206c77514b9c8b8fd56f300"}
Oct 30 19:49:21.935103 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.935077    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2" event={"ID":"aa47487f-ee16-4984-9091-1f277d564af2","Type":"ContainerDied","Data":"ecf6aae3fe9eea240cde0ea98641112a5ab1c9f23d8bb676a2672b485d5ca347"}
Oct 30 19:49:21.935103 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.935101    4712 scope.go:122] "RemoveContainer" containerID="d416c951ebbb5acb038fe7f0f5eb253a3404a3a81206c77514b9c8b8fd56f300"
Oct 30 19:49:21.944115 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.944099    4712 scope.go:122] "RemoveContainer" containerID="d416c951ebbb5acb038fe7f0f5eb253a3404a3a81206c77514b9c8b8fd56f300"
Oct 30 19:49:21.944419 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:49:21.944400    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d416c951ebbb5acb038fe7f0f5eb253a3404a3a81206c77514b9c8b8fd56f300\": container with ID starting with d416c951ebbb5acb038fe7f0f5eb253a3404a3a81206c77514b9c8b8fd56f300 not found: ID does not exist" containerID="d416c951ebbb5acb038fe7f0f5eb253a3404a3a81206c77514b9c8b8fd56f300"
Oct 30 19:49:21.944467 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.944436    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d416c951ebbb5acb038fe7f0f5eb253a3404a3a81206c77514b9c8b8fd56f300"} err="failed to get container status \"d416c951ebbb5acb038fe7f0f5eb253a3404a3a81206c77514b9c8b8fd56f300\": rpc error: code = NotFound desc = could not find container \"d416c951ebbb5acb038fe7f0f5eb253a3404a3a81206c77514b9c8b8fd56f300\": container with ID starting with d416c951ebbb5acb038fe7f0f5eb253a3404a3a81206c77514b9c8b8fd56f300 not found: ID does not exist"
Oct 30 19:49:21.986975 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.986952    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"]
Oct 30 19:49:21.999057 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:21.999038    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-544fc8-7944cd8b88-sdxl2"]
Oct 30 19:49:22.193641 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:22.193575    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:22.852455 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:22.852423    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa47487f-ee16-4984-9091-1f277d564af2" path="/var/lib/kubelet/pods/aa47487f-ee16-4984-9091-1f277d564af2/volumes"
Oct 30 19:49:28.296333 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:28.296296    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:28.472746 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:28.472714    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:28.598339 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:28.598256    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"]
Oct 30 19:49:28.598864 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:28.598832    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd" podUID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerName="istio-proxy" containerID="cri-o://b9527213d72efc1639e36c1981b3bfb56dd0473067d7813df820fd7a6242fd64" gracePeriod=30
Oct 30 19:49:28.678098 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:28.678069    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:30.643979 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:30.643944    4712 patch_prober.go:29] interesting pod/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:49:30.644341 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:30.643998    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd" podUID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:49:33.572619 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.572584    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:33.886616 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.886582    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:49:33.994973 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.994937    4712 generic.go:356] "Generic (PLEG): container finished" podID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerID="b9527213d72efc1639e36c1981b3bfb56dd0473067d7813df820fd7a6242fd64" exitCode=0
Oct 30 19:49:33.995171 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.994999    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"
Oct 30 19:49:33.995171 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.995050    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd" event={"ID":"d6ddfa89-9f31-4674-a41e-bcc43761b954","Type":"ContainerDied","Data":"b9527213d72efc1639e36c1981b3bfb56dd0473067d7813df820fd7a6242fd64"}
Oct 30 19:49:33.995171 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.995137    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd" event={"ID":"d6ddfa89-9f31-4674-a41e-bcc43761b954","Type":"ContainerDied","Data":"fdeb00c97ae0e65d4941020bcec63a8ecc7a03dcd5dfc10c239af029a58a8d54"}
Oct 30 19:49:33.995388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.995185    4712 scope.go:122] "RemoveContainer" containerID="b9527213d72efc1639e36c1981b3bfb56dd0473067d7813df820fd7a6242fd64"
Oct 30 19:49:33.995388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.995253    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-podinfo\") pod \"d6ddfa89-9f31-4674-a41e-bcc43761b954\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") "
Oct 30 19:49:33.995388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.995296    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-token\" (UniqueName: \"kubernetes.io/projected/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-token\") pod \"d6ddfa89-9f31-4674-a41e-bcc43761b954\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") "
Oct 30 19:49:33.995388 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.995354    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ddfa89-9f31-4674-a41e-bcc43761b954-kube-api-access-7nbtr\" (UniqueName: \"kubernetes.io/projected/d6ddfa89-9f31-4674-a41e-bcc43761b954-kube-api-access-7nbtr\") pod \"d6ddfa89-9f31-4674-a41e-bcc43761b954\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") "
Oct 30 19:49:33.995684 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.995432    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-workload-socket\") pod \"d6ddfa89-9f31-4674-a41e-bcc43761b954\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") "
Oct 30 19:49:33.995684 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.995466    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-data\") pod \"d6ddfa89-9f31-4674-a41e-bcc43761b954\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") "
Oct 30 19:49:33.995684 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.995506    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6ddfa89-9f31-4674-a41e-bcc43761b954-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ddfa89-9f31-4674-a41e-bcc43761b954-istiod-ca-cert\") pod \"d6ddfa89-9f31-4674-a41e-bcc43761b954\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") "
Oct 30 19:49:33.995684 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.995559    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-envoy\") pod \"d6ddfa89-9f31-4674-a41e-bcc43761b954\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") "
Oct 30 19:49:33.995684 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.995593    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-credential-socket\") pod \"d6ddfa89-9f31-4674-a41e-bcc43761b954\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") "
Oct 30 19:49:33.995684 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.995636    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-workload-certs\") pod \"d6ddfa89-9f31-4674-a41e-bcc43761b954\" (UID: \"d6ddfa89-9f31-4674-a41e-bcc43761b954\") "
Oct 30 19:49:33.996740 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.996378    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-data" pod "d6ddfa89-9f31-4674-a41e-bcc43761b954" (UID: "d6ddfa89-9f31-4674-a41e-bcc43761b954"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:33.997539 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.997502    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ddfa89-9f31-4674-a41e-bcc43761b954-istiod-ca-cert" pod "d6ddfa89-9f31-4674-a41e-bcc43761b954" (UID: "d6ddfa89-9f31-4674-a41e-bcc43761b954"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:33.997731 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.997703    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-credential-socket" pod "d6ddfa89-9f31-4674-a41e-bcc43761b954" (UID: "d6ddfa89-9f31-4674-a41e-bcc43761b954"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:33.997816 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.997779    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-workload-socket" pod "d6ddfa89-9f31-4674-a41e-bcc43761b954" (UID: "d6ddfa89-9f31-4674-a41e-bcc43761b954"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:33.998389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.998332    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-workload-certs" pod "d6ddfa89-9f31-4674-a41e-bcc43761b954" (UID: "d6ddfa89-9f31-4674-a41e-bcc43761b954"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:33.998801 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:33.998777    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-envoy" pod "d6ddfa89-9f31-4674-a41e-bcc43761b954" (UID: "d6ddfa89-9f31-4674-a41e-bcc43761b954"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:34.001405 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.000248    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ddfa89-9f31-4674-a41e-bcc43761b954-kube-api-access-7nbtr" pod "d6ddfa89-9f31-4674-a41e-bcc43761b954" (UID: "d6ddfa89-9f31-4674-a41e-bcc43761b954"). InnerVolumeSpecName "kube-api-access-7nbtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:34.001727 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.001705    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-token" pod "d6ddfa89-9f31-4674-a41e-bcc43761b954" (UID: "d6ddfa89-9f31-4674-a41e-bcc43761b954"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:34.002042 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.002014    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-podinfo" pod "d6ddfa89-9f31-4674-a41e-bcc43761b954" (UID: "d6ddfa89-9f31-4674-a41e-bcc43761b954"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:34.029577 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.029551    4712 scope.go:122] "RemoveContainer" containerID="1365a70761faa306c574957470df665ddeafe740d8d77e01d5f213c474d5f470"
Oct 30 19:49:34.041789 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.041757    4712 scope.go:122] "RemoveContainer" containerID="b9527213d72efc1639e36c1981b3bfb56dd0473067d7813df820fd7a6242fd64"
Oct 30 19:49:34.042154 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:49:34.042128    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9527213d72efc1639e36c1981b3bfb56dd0473067d7813df820fd7a6242fd64\": container with ID starting with b9527213d72efc1639e36c1981b3bfb56dd0473067d7813df820fd7a6242fd64 not found: ID does not exist" containerID="b9527213d72efc1639e36c1981b3bfb56dd0473067d7813df820fd7a6242fd64"
Oct 30 19:49:34.042266 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.042163    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9527213d72efc1639e36c1981b3bfb56dd0473067d7813df820fd7a6242fd64"} err="failed to get container status \"b9527213d72efc1639e36c1981b3bfb56dd0473067d7813df820fd7a6242fd64\": rpc error: code = NotFound desc = could not find container \"b9527213d72efc1639e36c1981b3bfb56dd0473067d7813df820fd7a6242fd64\": container with ID starting with b9527213d72efc1639e36c1981b3bfb56dd0473067d7813df820fd7a6242fd64 not found: ID does not exist"
Oct 30 19:49:34.042266 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.042181    4712 scope.go:122] "RemoveContainer" containerID="1365a70761faa306c574957470df665ddeafe740d8d77e01d5f213c474d5f470"
Oct 30 19:49:34.042526 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:49:34.042507    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1365a70761faa306c574957470df665ddeafe740d8d77e01d5f213c474d5f470\": container with ID starting with 1365a70761faa306c574957470df665ddeafe740d8d77e01d5f213c474d5f470 not found: ID does not exist" containerID="1365a70761faa306c574957470df665ddeafe740d8d77e01d5f213c474d5f470"
Oct 30 19:49:34.042627 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.042539    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1365a70761faa306c574957470df665ddeafe740d8d77e01d5f213c474d5f470"} err="failed to get container status \"1365a70761faa306c574957470df665ddeafe740d8d77e01d5f213c474d5f470\": rpc error: code = NotFound desc = could not find container \"1365a70761faa306c574957470df665ddeafe740d8d77e01d5f213c474d5f470\": container with ID starting with 1365a70761faa306c574957470df665ddeafe740d8d77e01d5f213c474d5f470 not found: ID does not exist"
Oct 30 19:49:34.097195 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.097166    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:34.097274 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.097201    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nbtr\" (UniqueName: \"kubernetes.io/projected/d6ddfa89-9f31-4674-a41e-bcc43761b954-kube-api-access-7nbtr\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:34.097274 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.097217    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:34.097274 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.097231    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:34.097274 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.097245    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6ddfa89-9f31-4674-a41e-bcc43761b954-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:34.097274 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.097257    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:34.097274 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.097270    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:34.097509 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.097286    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6ddfa89-9f31-4674-a41e-bcc43761b954-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:34.097509 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.097299    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6ddfa89-9f31-4674-a41e-bcc43761b954-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:49:34.345861 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.345832    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"]
Oct 30 19:49:34.357760 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.357735    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-544fc8-openshift-default-6b56b8b795-2hghd"]
Oct 30 19:49:34.875081 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:34.875044    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6ddfa89-9f31-4674-a41e-bcc43761b954" path="/var/lib/kubelet/pods/d6ddfa89-9f31-4674-a41e-bcc43761b954/volumes"
Oct 30 19:49:36.379179 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:49:36.379132    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:49:36.701485 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:36.701405    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:36.936141 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:36.936099    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:37.134786 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:37.134760    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:37.396347 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:37.396272    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:37.633437 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:37.633407    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:43.393785 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:43.393677    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:44.673913 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:44.673882    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:44.913346 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:44.913312    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:48.510269 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:48.510236    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:48.651396 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:48.651351    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:49:54.075309 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:49:54.075275    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:50:06.372612 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:50:06.372574    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:50:08.908772 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:50:08.908739    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:50:10.597308 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:50:10.597280    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:50:33.707139 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:50:33.707100    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:50:36.373104 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:50:36.373058    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:50:46.172192 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:50:46.172153    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:50:46.334052 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:50:46.334020    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:50:47.088298 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:50:47.088265    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:50:47.092477 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:50:47.092455    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:51:06.371891 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:51:06.371855    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:51:21.755029 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:51:21.754995    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:51:33.765208 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:51:33.765134    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:51:36.373346 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:51:36.373309    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:51:44.727330 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:51:44.727296    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:51:45.092945 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:51:45.092858    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:51:48.010620 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:51:48.010588    4712 scope.go:122] "RemoveContainer" containerID="2f21c5b8bc6c33bd33414588545bd366f0b01530e5f1568ffef19c580d83f923"
Oct 30 19:52:06.373302 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:52:06.373261    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:52:14.558189 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:14.558154    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:52:14.704486 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:14.704455    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:52:27.968679 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:27.968615    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:52:28.132650 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:28.132618    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:52:33.877577 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:33.877543    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:52:36.372009 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:52:36.371968    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:52:38.428093 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.428062    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"]
Oct 30 19:52:38.428562 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.428273    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44" podUID="6785a121-64c7-4514-8e16-aeba1d89ae02" containerName="discovery" containerID="cri-o://d50f678a81bd11d411f10c002fafedadeb8e36a2fd9bc1beeb0773051e107732" gracePeriod=30
Oct 30 19:52:38.661302 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.661277    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:52:38.841491 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.841422    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6785a121-64c7-4514-8e16-aeba1d89ae02-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6785a121-64c7-4514-8e16-aeba1d89ae02-local-certs\") pod \"6785a121-64c7-4514-8e16-aeba1d89ae02\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") "
Oct 30 19:52:38.841491 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.841473    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-cacerts\" (UniqueName: \"kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-cacerts\") pod \"6785a121-64c7-4514-8e16-aeba1d89ae02\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") "
Oct 30 19:52:38.841687 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.841504    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-token\" (UniqueName: \"kubernetes.io/projected/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-token\") pod \"6785a121-64c7-4514-8e16-aeba1d89ae02\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") "
Oct 30 19:52:38.841687 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.841557    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-kubeconfig\") pod \"6785a121-64c7-4514-8e16-aeba1d89ae02\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") "
Oct 30 19:52:38.841687 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.841632    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-csr-dns-cert\") pod \"6785a121-64c7-4514-8e16-aeba1d89ae02\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") "
Oct 30 19:52:38.841687 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.841668    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-csr-ca-configmap\") pod \"6785a121-64c7-4514-8e16-aeba1d89ae02\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") "
Oct 30 19:52:38.841915 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.841698    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6785a121-64c7-4514-8e16-aeba1d89ae02-kube-api-access-d55z8\" (UniqueName: \"kubernetes.io/projected/6785a121-64c7-4514-8e16-aeba1d89ae02-kube-api-access-d55z8\") pod \"6785a121-64c7-4514-8e16-aeba1d89ae02\" (UID: \"6785a121-64c7-4514-8e16-aeba1d89ae02\") "
Oct 30 19:52:38.842119 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.842078    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-csr-ca-configmap" pod "6785a121-64c7-4514-8e16-aeba1d89ae02" (UID: "6785a121-64c7-4514-8e16-aeba1d89ae02"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:38.843722 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.843684    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6785a121-64c7-4514-8e16-aeba1d89ae02-local-certs" pod "6785a121-64c7-4514-8e16-aeba1d89ae02" (UID: "6785a121-64c7-4514-8e16-aeba1d89ae02"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:38.844270 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.844243    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-kubeconfig" pod "6785a121-64c7-4514-8e16-aeba1d89ae02" (UID: "6785a121-64c7-4514-8e16-aeba1d89ae02"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:38.844270 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.844260    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-token" pod "6785a121-64c7-4514-8e16-aeba1d89ae02" (UID: "6785a121-64c7-4514-8e16-aeba1d89ae02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:38.844441 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.844253    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-cacerts" pod "6785a121-64c7-4514-8e16-aeba1d89ae02" (UID: "6785a121-64c7-4514-8e16-aeba1d89ae02"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:38.844441 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.844275    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-csr-dns-cert" pod "6785a121-64c7-4514-8e16-aeba1d89ae02" (UID: "6785a121-64c7-4514-8e16-aeba1d89ae02"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:52:38.844441 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.844277    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6785a121-64c7-4514-8e16-aeba1d89ae02-kube-api-access-d55z8" pod "6785a121-64c7-4514-8e16-aeba1d89ae02" (UID: "6785a121-64c7-4514-8e16-aeba1d89ae02"). InnerVolumeSpecName "kube-api-access-d55z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:38.879185 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.879155    4712 generic.go:356] "Generic (PLEG): container finished" podID="6785a121-64c7-4514-8e16-aeba1d89ae02" containerID="d50f678a81bd11d411f10c002fafedadeb8e36a2fd9bc1beeb0773051e107732" exitCode=0
Oct 30 19:52:38.879297 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.879195    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"
Oct 30 19:52:38.879297 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.879231    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44" event={"ID":"6785a121-64c7-4514-8e16-aeba1d89ae02","Type":"ContainerDied","Data":"d50f678a81bd11d411f10c002fafedadeb8e36a2fd9bc1beeb0773051e107732"}
Oct 30 19:52:38.879297 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.879277    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44" event={"ID":"6785a121-64c7-4514-8e16-aeba1d89ae02","Type":"ContainerDied","Data":"740cf49b119e248b220ec891e59a5bd990bba25d1da106891b2f711ae7feb868"}
Oct 30 19:52:38.879297 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.879298    4712 scope.go:122] "RemoveContainer" containerID="d50f678a81bd11d411f10c002fafedadeb8e36a2fd9bc1beeb0773051e107732"
Oct 30 19:52:38.887777 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.887749    4712 scope.go:122] "RemoveContainer" containerID="d50f678a81bd11d411f10c002fafedadeb8e36a2fd9bc1beeb0773051e107732"
Oct 30 19:52:38.888022 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:52:38.888006    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d50f678a81bd11d411f10c002fafedadeb8e36a2fd9bc1beeb0773051e107732\": container with ID starting with d50f678a81bd11d411f10c002fafedadeb8e36a2fd9bc1beeb0773051e107732 not found: ID does not exist" containerID="d50f678a81bd11d411f10c002fafedadeb8e36a2fd9bc1beeb0773051e107732"
Oct 30 19:52:38.888073 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.888029    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50f678a81bd11d411f10c002fafedadeb8e36a2fd9bc1beeb0773051e107732"} err="failed to get container status \"d50f678a81bd11d411f10c002fafedadeb8e36a2fd9bc1beeb0773051e107732\": rpc error: code = NotFound desc = could not find container \"d50f678a81bd11d411f10c002fafedadeb8e36a2fd9bc1beeb0773051e107732\": container with ID starting with d50f678a81bd11d411f10c002fafedadeb8e36a2fd9bc1beeb0773051e107732 not found: ID does not exist"
Oct 30 19:52:38.929914 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.929884    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"]
Oct 30 19:52:38.936529 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.936508    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xqw44"]
Oct 30 19:52:38.943033 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.943013    4712 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6785a121-64c7-4514-8e16-aeba1d89ae02-local-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:52:38.943033 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.943034    4712 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-cacerts\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:52:38.943150 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.943042    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:52:38.943150 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.943054    4712 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-kubeconfig\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:52:38.943150 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.943066    4712 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-csr-dns-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:52:38.943150 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.943075    4712 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6785a121-64c7-4514-8e16-aeba1d89ae02-istio-csr-ca-configmap\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:52:38.943150 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:38.943085    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d55z8\" (UniqueName: \"kubernetes.io/projected/6785a121-64c7-4514-8e16-aeba1d89ae02-kube-api-access-d55z8\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:52:40.850257 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:40.850215    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6785a121-64c7-4514-8e16-aeba1d89ae02" path="/var/lib/kubelet/pods/6785a121-64c7-4514-8e16-aeba1d89ae02/volumes"
Oct 30 19:52:48.060386 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:48.057525    4712 scope.go:122] "RemoveContainer" containerID="140b8df38c6e519f68bd0c780ab509fde97e90c5633edffc4961c22c5dff2a4e"
Oct 30 19:52:48.419520 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:48.419453    4712 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-28vt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:48.419904 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:52:48.419861    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-28vt6" podUID="8cbf06ec-e967-4cc4-9d06-6629e3be62da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:06.372717 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:53:06.372642    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:53:07.777435 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:53:07.777390    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:53:34.017419 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:53:34.017383    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:53:36.371918 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:53:36.371883    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:53:40.251048 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:53:40.251011    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:53:40.400077 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:53:40.400045    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:53:41.017631 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:53:41.017548    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:53:41.277140 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:53:41.277056    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:06.371894 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:54:06.371858    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:54:09.515269 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:54:09.515234    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:09.675510 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:54:09.675474    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:34.127453 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:54:34.127349    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:54:36.372042 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:54:36.372009    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:55:06.372387 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:55:06.372333    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:55:13.383501 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:13.383459    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:15.095852 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.095822    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:15.456673 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.456641    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"]
Oct 30 19:55:15.457230 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457215    4712 cpu_manager.go:414] "containerMap: removing container" podUID="3fce9cac-38e0-436f-a4aa-4f82b908e11e" containerName="istio-proxy"
Oct 30 19:55:15.457311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457231    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fce9cac-38e0-436f-a4aa-4f82b908e11e" containerName="istio-proxy"
Oct 30 19:55:15.457311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457243    4712 cpu_manager.go:414] "containerMap: removing container" podUID="045af806-47d7-41b6-9cd7-ab0b9c851f3c" containerName="istio-proxy"
Oct 30 19:55:15.457311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457248    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045af806-47d7-41b6-9cd7-ab0b9c851f3c" containerName="istio-proxy"
Oct 30 19:55:15.457311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457263    4712 cpu_manager.go:414] "containerMap: removing container" podUID="33d1922a-a6fa-4a8e-8148-174f2ef400af" containerName="istio-proxy"
Oct 30 19:55:15.457311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457268    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33d1922a-a6fa-4a8e-8148-174f2ef400af" containerName="istio-proxy"
Oct 30 19:55:15.457311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457276    4712 cpu_manager.go:414] "containerMap: removing container" podUID="bb85c12b-5f17-4e75-9a68-96e9f83389b1" containerName="istio-proxy"
Oct 30 19:55:15.457311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457281    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb85c12b-5f17-4e75-9a68-96e9f83389b1" containerName="istio-proxy"
Oct 30 19:55:15.457311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457294    4712 cpu_manager.go:414] "containerMap: removing container" podUID="6785a121-64c7-4514-8e16-aeba1d89ae02" containerName="discovery"
Oct 30 19:55:15.457311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457299    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6785a121-64c7-4514-8e16-aeba1d89ae02" containerName="discovery"
Oct 30 19:55:15.457311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457306    4712 cpu_manager.go:414] "containerMap: removing container" podUID="35cbcb48-e970-48e6-a942-05cd29730d9a" containerName="istio-proxy"
Oct 30 19:55:15.457311 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457313    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35cbcb48-e970-48e6-a942-05cd29730d9a" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457326    4712 cpu_manager.go:414] "containerMap: removing container" podUID="82a18b97-a6fc-4f33-848b-86bb8ef66fc1" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457331    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a18b97-a6fc-4f33-848b-86bb8ef66fc1" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457338    4712 cpu_manager.go:414] "containerMap: removing container" podUID="b727df7a-56c7-46fc-974a-38aebc2593c3" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457343    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b727df7a-56c7-46fc-974a-38aebc2593c3" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457355    4712 cpu_manager.go:414] "containerMap: removing container" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457374    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457387    4712 cpu_manager.go:414] "containerMap: removing container" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457396    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457403    4712 cpu_manager.go:414] "containerMap: removing container" podUID="b4ea703f-3d59-4204-9630-e5915516c074" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457407    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ea703f-3d59-4204-9630-e5915516c074" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457414    4712 cpu_manager.go:414] "containerMap: removing container" podUID="d19408be-2ea5-4445-99a5-9094c497b95f" containerName="payload-processing"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457418    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d19408be-2ea5-4445-99a5-9094c497b95f" containerName="payload-processing"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457433    4712 cpu_manager.go:414] "containerMap: removing container" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457442    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457449    4712 cpu_manager.go:414] "containerMap: removing container" podUID="d2fa6bc2-624f-4017-898f-2317c9064c73" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457453    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2fa6bc2-624f-4017-898f-2317c9064c73" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457465    4712 cpu_manager.go:414] "containerMap: removing container" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457470    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457482    4712 cpu_manager.go:414] "containerMap: removing container" podUID="d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457513    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457521    4712 cpu_manager.go:414] "containerMap: removing container" podUID="804735e2-421e-40c5-8254-b83c9171ce7c" containerName="revoke-keys"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457526    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="804735e2-421e-40c5-8254-b83c9171ce7c" containerName="revoke-keys"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457541    4712 cpu_manager.go:414] "containerMap: removing container" podUID="700a4ae0-505b-4926-93c0-ee5b5f627030" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457548    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700a4ae0-505b-4926-93c0-ee5b5f627030" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457562    4712 cpu_manager.go:414] "containerMap: removing container" podUID="aa47487f-ee16-4984-9091-1f277d564af2" containerName="maas-api"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457567    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa47487f-ee16-4984-9091-1f277d564af2" containerName="maas-api"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457574    4712 cpu_manager.go:414] "containerMap: removing container" podUID="e42358a7-0f5b-476a-831a-cefabde3984e" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457579    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42358a7-0f5b-476a-831a-cefabde3984e" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457591    4712 cpu_manager.go:414] "containerMap: removing container" podUID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457596    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457603    4712 cpu_manager.go:414] "containerMap: removing container" podUID="62d985a2-697f-4a97-a50b-fdecff1d5960" containerName="payload-pre-processing"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457607    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d985a2-697f-4a97-a50b-fdecff1d5960" containerName="payload-pre-processing"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457621    4712 cpu_manager.go:414] "containerMap: removing container" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457625    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457638    4712 cpu_manager.go:414] "containerMap: removing container" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457643    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457657    4712 cpu_manager.go:414] "containerMap: removing container" podUID="27c8e215-0d99-44f8-88cd-8c5fe3b964b1" containerName="maas-api"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457677    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c8e215-0d99-44f8-88cd-8c5fe3b964b1" containerName="maas-api"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457687    4712 cpu_manager.go:414] "containerMap: removing container" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457692    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457707    4712 cpu_manager.go:414] "containerMap: removing container" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457712    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457724    4712 cpu_manager.go:414] "containerMap: removing container" podUID="b8c1988d-3a60-4e33-b9f2-c9878beb006b" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457728    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c1988d-3a60-4e33-b9f2-c9878beb006b" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457740    4712 cpu_manager.go:414] "containerMap: removing container" podUID="3d36e5e5-7f21-45a4-961f-61196c8a5e3e" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457744    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d36e5e5-7f21-45a4-961f-61196c8a5e3e" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457752    4712 cpu_manager.go:414] "containerMap: removing container" podUID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457756    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457769    4712 cpu_manager.go:414] "containerMap: removing container" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457773    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457781    4712 cpu_manager.go:414] "containerMap: removing container" podUID="b4ea703f-3d59-4204-9630-e5915516c074" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457785    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ea703f-3d59-4204-9630-e5915516c074" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457796    4712 cpu_manager.go:414] "containerMap: removing container" podUID="245c3fb3-0b83-4d71-be3c-81d6cafb639c" containerName="payload-pre-processing"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457801    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="245c3fb3-0b83-4d71-be3c-81d6cafb639c" containerName="payload-pre-processing"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457808    4712 cpu_manager.go:414] "containerMap: removing container" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457813    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457823    4712 cpu_manager.go:414] "containerMap: removing container" podUID="b727df7a-56c7-46fc-974a-38aebc2593c3" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457829    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b727df7a-56c7-46fc-974a-38aebc2593c3" containerName="istio-proxy"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457836    4712 cpu_manager.go:414] "containerMap: removing container" podUID="60db5e44-82df-473c-9fc5-f7682f2c7000" containerName="maas-api"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457840    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60db5e44-82df-473c-9fc5-f7682f2c7000" containerName="maas-api"
Oct 30 19:55:15.457815 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457852    4712 cpu_manager.go:414] "containerMap: removing container" podUID="fd990c24-235b-4f45-92a1-da3e7c443153" containerName="payload-processing"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457857    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd990c24-235b-4f45-92a1-da3e7c443153" containerName="payload-processing"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457864    4712 cpu_manager.go:414] "containerMap: removing container" podUID="ced985e1-6978-4dcb-8cc9-7312fe9a16d7" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457885    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced985e1-6978-4dcb-8cc9-7312fe9a16d7" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457896    4712 cpu_manager.go:414] "containerMap: removing container" podUID="7f364e92-b446-4671-8a42-573ebe98d729" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457902    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f364e92-b446-4671-8a42-573ebe98d729" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457913    4712 cpu_manager.go:414] "containerMap: removing container" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457917    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457930    4712 cpu_manager.go:414] "containerMap: removing container" podUID="b3cd29f0-6462-40b1-a5a0-30deec5e9321" containerName="maas-api"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.457935    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3cd29f0-6462-40b1-a5a0-30deec5e9321" containerName="maas-api"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458054    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="804735e2-421e-40c5-8254-b83c9171ce7c" containerName="revoke-keys"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458068    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="700a4ae0-505b-4926-93c0-ee5b5f627030" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458075    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa47487f-ee16-4984-9091-1f277d564af2" containerName="maas-api"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458081    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42358a7-0f5b-476a-831a-cefabde3984e" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458091    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458100    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d985a2-697f-4a97-a50b-fdecff1d5960" containerName="payload-pre-processing"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458107    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a3b2eb-8722-4782-af07-70239b3c0ff5" containerName="main"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458111    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d39453-861c-437a-8bbe-e9942b8778d3" containerName="main"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458121    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c8e215-0d99-44f8-88cd-8c5fe3b964b1" containerName="maas-api"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458132    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3bee1cf-6278-4666-93db-a483e7dca394" containerName="main"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458138    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc58f78-58ef-4981-b1a9-4c6aa8b781fc" containerName="main"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458143    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c1988d-3a60-4e33-b9f2-c9878beb006b" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458152    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d36e5e5-7f21-45a4-961f-61196c8a5e3e" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458157    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ddfa89-9f31-4674-a41e-bcc43761b954" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458167    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e3a8512-59ff-45bf-b7c7-55369c05d8ce" containerName="main"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458186    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="35cbcb48-e970-48e6-a942-05cd29730d9a" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458194    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ea703f-3d59-4204-9630-e5915516c074" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458202    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="245c3fb3-0b83-4d71-be3c-81d6cafb639c" containerName="payload-pre-processing"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458208    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a6dccf3-9a50-43ef-833f-1e66fe53890f" containerName="main"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458217    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="b727df7a-56c7-46fc-974a-38aebc2593c3" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458222    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fce9cac-38e0-436f-a4aa-4f82b908e11e" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458227    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="60db5e44-82df-473c-9fc5-f7682f2c7000" containerName="maas-api"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458237    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="045af806-47d7-41b6-9cd7-ab0b9c851f3c" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458243    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd990c24-235b-4f45-92a1-da3e7c443153" containerName="payload-processing"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458247    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced985e1-6978-4dcb-8cc9-7312fe9a16d7" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458252    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f364e92-b446-4671-8a42-573ebe98d729" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458263    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d5b13af-b0c4-4343-80ca-5c5e0d0c769f" containerName="main"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458272    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3cd29f0-6462-40b1-a5a0-30deec5e9321" containerName="maas-api"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458278    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fce9cac-38e0-436f-a4aa-4f82b908e11e" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458288    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="33d1922a-a6fa-4a8e-8148-174f2ef400af" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458294    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb85c12b-5f17-4e75-9a68-96e9f83389b1" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458300    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="6785a121-64c7-4514-8e16-aeba1d89ae02" containerName="discovery"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458309    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="35cbcb48-e970-48e6-a942-05cd29730d9a" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458315    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a18b97-a6fc-4f33-848b-86bb8ef66fc1" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458324    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="b727df7a-56c7-46fc-974a-38aebc2593c3" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458331    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a486ae6-e874-4317-ab9b-e6cdebc7ce89" containerName="main"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458340    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="946e2596-1023-4d05-ba84-2dd3da74cc8e" containerName="main"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458345    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ea703f-3d59-4204-9630-e5915516c074" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458355    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="d19408be-2ea5-4445-99a5-9094c497b95f" containerName="payload-processing"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458373    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa037781-1c62-492d-adcc-006600af37ff" containerName="main"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458385    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2fa6bc2-624f-4017-898f-2317c9064c73" containerName="istio-proxy"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458390    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="e640ac87-e1bd-4f61-8848-f82dd4835442" containerName="main"
Oct 30 19:55:15.460483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.458397    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91c1dde-2ad0-4d1e-9846-6d7db14fc6f2" containerName="istio-proxy"
Oct 30 19:55:15.470890 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.470865    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.480646 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.480618    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"]
Oct 30 19:55:15.490735 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.490711    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-1f3297-openshift-default-dockercfg-4k26q\""
Oct 30 19:55:15.562737 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.562708    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:15.616948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.616918    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-workload-certs\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.617069 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.617022    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-credential-socket\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.617172 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.617147    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ece469f-27b4-48f2-b640-86d165397d23-istiod-ca-cert\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.617212 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.617190    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff2wn\" (UniqueName: \"kubernetes.io/projected/4ece469f-27b4-48f2-b640-86d165397d23-kube-api-access-ff2wn\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.617347 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.617321    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ece469f-27b4-48f2-b640-86d165397d23-istio-token\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.617495 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.617397    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ece469f-27b4-48f2-b640-86d165397d23-istio-podinfo\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.617495 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.617432    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-istio-envoy\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.617495 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.617450    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-istio-data\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.617756 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.617540    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-workload-socket\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.712811 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.712730    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:15.718977 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.718949    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-workload-certs\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.719118 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.718990    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-credential-socket\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.719118 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.719040    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ece469f-27b4-48f2-b640-86d165397d23-istiod-ca-cert\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.719118 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.719064    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff2wn\" (UniqueName: \"kubernetes.io/projected/4ece469f-27b4-48f2-b640-86d165397d23-kube-api-access-ff2wn\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.719287 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.719123    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ece469f-27b4-48f2-b640-86d165397d23-istio-token\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.719287 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.719163    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ece469f-27b4-48f2-b640-86d165397d23-istio-podinfo\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.719287 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.719203    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-istio-envoy\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.719287 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.719232    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-istio-data\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.719287 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.719269    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-workload-socket\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.719580 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.719404    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-credential-socket\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.719674 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.719649    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-workload-socket\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.719739 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.719698    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-workload-certs\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.719739 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.719714    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-istio-data\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.719822 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.719801    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ece469f-27b4-48f2-b640-86d165397d23-istiod-ca-cert\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.721464 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.721444    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-istio-envoy\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.721774 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.721751    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ece469f-27b4-48f2-b640-86d165397d23-istio-podinfo\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.739308 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.739286    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ece469f-27b4-48f2-b640-86d165397d23-istio-token\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.739459 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.739291    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff2wn\" (UniqueName: \"kubernetes.io/projected/4ece469f-27b4-48f2-b640-86d165397d23-kube-api-access-ff2wn\") pod \"e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") " pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:15.787664 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:15.787634    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:16.094659 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:16.094635    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"]
Oct 30 19:55:16.096602 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:55:16.096577    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ece469f_27b4_48f2_b640_86d165397d23.slice/crio-52ab1abb37c700352547d979bb763611f1f1e97b5542182f06b54459a145af9b WatchSource:0}: Error finding container 52ab1abb37c700352547d979bb763611f1f1e97b5542182f06b54459a145af9b: Status 404 returned error can't find the container with id 52ab1abb37c700352547d979bb763611f1f1e97b5542182f06b54459a145af9b
Oct 30 19:55:16.098710 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:16.098676    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:55:16.098782 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:16.098743    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:55:16.098782 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:16.098771    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:55:16.633622 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:16.633580    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c" event={"ID":"4ece469f-27b4-48f2-b640-86d165397d23","Type":"ContainerStarted","Data":"01fe9eb235d635d9864dccd8cab21df03dbc4f898c5f7f65ef8d783387e38477"}
Oct 30 19:55:16.633819 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:16.633649    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c" event={"ID":"4ece469f-27b4-48f2-b640-86d165397d23","Type":"ContainerStarted","Data":"52ab1abb37c700352547d979bb763611f1f1e97b5542182f06b54459a145af9b"}
Oct 30 19:55:16.676314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:16.676260    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c" podStartSLOduration=1.676248183 podStartE2EDuration="1.676248183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:16.676018989 +0000 UTC m=+2069.975281120" watchObservedRunningTime="2026-10-30 19:55:16.676248183 +0000 UTC m=+2069.975510311"
Oct 30 19:55:16.787838 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:16.787806    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:17.793258 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:17.793230    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:18.642885 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:18.642856    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:18.643721 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:18.643702    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:19.152702 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:19.152669    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:19.299282 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:19.299247    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:20.439125 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.439093    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"]
Oct 30 19:55:20.439537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.439474    4712 cpu_manager.go:414] "containerMap: removing container" podUID="045af806-47d7-41b6-9cd7-ab0b9c851f3c" containerName="istio-proxy"
Oct 30 19:55:20.439537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.439489    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045af806-47d7-41b6-9cd7-ab0b9c851f3c" containerName="istio-proxy"
Oct 30 19:55:20.439537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.439501    4712 cpu_manager.go:414] "containerMap: removing container" podUID="35cbcb48-e970-48e6-a942-05cd29730d9a" containerName="istio-proxy"
Oct 30 19:55:20.439537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.439506    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35cbcb48-e970-48e6-a942-05cd29730d9a" containerName="istio-proxy"
Oct 30 19:55:20.439537 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.439538    4712 cpu_manager.go:414] "containerMap: removing container" podUID="3fce9cac-38e0-436f-a4aa-4f82b908e11e" containerName="istio-proxy"
Oct 30 19:55:20.439702 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.439544    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fce9cac-38e0-436f-a4aa-4f82b908e11e" containerName="istio-proxy"
Oct 30 19:55:20.439702 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.439602    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="045af806-47d7-41b6-9cd7-ab0b9c851f3c" containerName="istio-proxy"
Oct 30 19:55:20.442857 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.442839    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"
Oct 30 19:55:20.472709 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.472683    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-1f3297-serving-cert\""
Oct 30 19:55:20.475552 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.475525    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"]
Oct 30 19:55:20.557708 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.557674    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/520ed64e-c5f4-4a6c-b11a-dc5850632383-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz\" (UID: \"520ed64e-c5f4-4a6c-b11a-dc5850632383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"
Oct 30 19:55:20.557878 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.557773    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/520ed64e-c5f4-4a6c-b11a-dc5850632383-metrics-certs\") pod \"maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz\" (UID: \"520ed64e-c5f4-4a6c-b11a-dc5850632383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"
Oct 30 19:55:20.557878 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.557820    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/520ed64e-c5f4-4a6c-b11a-dc5850632383-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz\" (UID: \"520ed64e-c5f4-4a6c-b11a-dc5850632383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"
Oct 30 19:55:20.557981 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.557943    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9vd4\" (UniqueName: \"kubernetes.io/projected/520ed64e-c5f4-4a6c-b11a-dc5850632383-kube-api-access-f9vd4\") pod \"maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz\" (UID: \"520ed64e-c5f4-4a6c-b11a-dc5850632383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"
Oct 30 19:55:20.610314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.610282    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:20.658414 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.658375    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9vd4\" (UniqueName: \"kubernetes.io/projected/520ed64e-c5f4-4a6c-b11a-dc5850632383-kube-api-access-f9vd4\") pod \"maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz\" (UID: \"520ed64e-c5f4-4a6c-b11a-dc5850632383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"
Oct 30 19:55:20.658608 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.658459    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/520ed64e-c5f4-4a6c-b11a-dc5850632383-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz\" (UID: \"520ed64e-c5f4-4a6c-b11a-dc5850632383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"
Oct 30 19:55:20.658608 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.658487    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/520ed64e-c5f4-4a6c-b11a-dc5850632383-metrics-certs\") pod \"maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz\" (UID: \"520ed64e-c5f4-4a6c-b11a-dc5850632383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"
Oct 30 19:55:20.658608 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.658524    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/520ed64e-c5f4-4a6c-b11a-dc5850632383-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz\" (UID: \"520ed64e-c5f4-4a6c-b11a-dc5850632383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"
Oct 30 19:55:20.659154 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.659132    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/520ed64e-c5f4-4a6c-b11a-dc5850632383-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz\" (UID: \"520ed64e-c5f4-4a6c-b11a-dc5850632383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"
Oct 30 19:55:20.660906 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.660884    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/520ed64e-c5f4-4a6c-b11a-dc5850632383-metrics-certs\") pod \"maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz\" (UID: \"520ed64e-c5f4-4a6c-b11a-dc5850632383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"
Oct 30 19:55:20.661008 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.660962    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/520ed64e-c5f4-4a6c-b11a-dc5850632383-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz\" (UID: \"520ed64e-c5f4-4a6c-b11a-dc5850632383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"
Oct 30 19:55:20.681154 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.681127    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9vd4\" (UniqueName: \"kubernetes.io/projected/520ed64e-c5f4-4a6c-b11a-dc5850632383-kube-api-access-f9vd4\") pod \"maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz\" (UID: \"520ed64e-c5f4-4a6c-b11a-dc5850632383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"
Oct 30 19:55:20.756467 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.756397    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"
Oct 30 19:55:20.777226 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.777188    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:20.914237 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.914202    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p"]
Oct 30 19:55:20.917727 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.917709    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p"
Oct 30 19:55:20.932489 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.932463    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-66dfadcf-dockercfg-9xn6m\""
Oct 30 19:55:20.940643 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.940619    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p"]
Oct 30 19:55:20.947128 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.947108    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:20.956166 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:20.956138    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-1f3297\""
Oct 30 19:55:21.007781 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.007701    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8"]
Oct 30 19:55:21.011589 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.011562    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8"
Oct 30 19:55:21.033761 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.033730    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8"]
Oct 30 19:55:21.062621 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.062589    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfrmk\" (UniqueName: \"kubernetes.io/projected/9af6c27c-16a0-4d18-a328-06b585f944eb-kube-api-access-rfrmk\") pod \"payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p\" (UID: \"9af6c27c-16a0-4d18-a328-06b585f944eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p"
Oct 30 19:55:21.062799 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.062630    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af6c27c-16a0-4d18-a328-06b585f944eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p\" (UID: \"9af6c27c-16a0-4d18-a328-06b585f944eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p"
Oct 30 19:55:21.076158 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.076137    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"]
Oct 30 19:55:21.076906 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:55:21.076885    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520ed64e_c5f4_4a6c_b11a_dc5850632383.slice/crio-5c315d095a38e456089b0a57f6a96068ce30725d7b9e3b49fd07521f910147bb WatchSource:0}: Error finding container 5c315d095a38e456089b0a57f6a96068ce30725d7b9e3b49fd07521f910147bb: Status 404 returned error can't find the container with id 5c315d095a38e456089b0a57f6a96068ce30725d7b9e3b49fd07521f910147bb
Oct 30 19:55:21.078330 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.078313    4712 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:55:21.163951 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.163920    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7540294-e7e5-42a5-8868-a7e518e7398a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8\" (UID: \"f7540294-e7e5-42a5-8868-a7e518e7398a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8"
Oct 30 19:55:21.164076 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.163971    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdfzz\" (UniqueName: \"kubernetes.io/projected/f7540294-e7e5-42a5-8868-a7e518e7398a-kube-api-access-hdfzz\") pod \"payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8\" (UID: \"f7540294-e7e5-42a5-8868-a7e518e7398a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8"
Oct 30 19:55:21.164076 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.164043    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfrmk\" (UniqueName: \"kubernetes.io/projected/9af6c27c-16a0-4d18-a328-06b585f944eb-kube-api-access-rfrmk\") pod \"payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p\" (UID: \"9af6c27c-16a0-4d18-a328-06b585f944eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p"
Oct 30 19:55:21.164076 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.164075    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af6c27c-16a0-4d18-a328-06b585f944eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p\" (UID: \"9af6c27c-16a0-4d18-a328-06b585f944eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p"
Oct 30 19:55:21.164773 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.164753    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af6c27c-16a0-4d18-a328-06b585f944eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p\" (UID: \"9af6c27c-16a0-4d18-a328-06b585f944eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p"
Oct 30 19:55:21.184528 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.184504    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfrmk\" (UniqueName: \"kubernetes.io/projected/9af6c27c-16a0-4d18-a328-06b585f944eb-kube-api-access-rfrmk\") pod \"payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p\" (UID: \"9af6c27c-16a0-4d18-a328-06b585f944eb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p"
Oct 30 19:55:21.230498 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.230477    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p"
Oct 30 19:55:21.266592 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.266509    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7540294-e7e5-42a5-8868-a7e518e7398a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8\" (UID: \"f7540294-e7e5-42a5-8868-a7e518e7398a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8"
Oct 30 19:55:21.266592 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.266580    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7540294-e7e5-42a5-8868-a7e518e7398a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8\" (UID: \"f7540294-e7e5-42a5-8868-a7e518e7398a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8"
Oct 30 19:55:21.266829 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.266657    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdfzz\" (UniqueName: \"kubernetes.io/projected/f7540294-e7e5-42a5-8868-a7e518e7398a-kube-api-access-hdfzz\") pod \"payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8\" (UID: \"f7540294-e7e5-42a5-8868-a7e518e7398a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8"
Oct 30 19:55:21.288029 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.287749    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdfzz\" (UniqueName: \"kubernetes.io/projected/f7540294-e7e5-42a5-8868-a7e518e7398a-kube-api-access-hdfzz\") pod \"payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8\" (UID: \"f7540294-e7e5-42a5-8868-a7e518e7398a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8"
Oct 30 19:55:21.327767 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.327741    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8"
Oct 30 19:55:21.530672 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.530628    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p"]
Oct 30 19:55:21.532001 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:55:21.531969    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af6c27c_16a0_4d18_a328_06b585f944eb.slice/crio-63ea9123ee976bac7a3ac8f4eaec3170019e877813b2fa6b4f6067cb53266c6d WatchSource:0}: Error finding container 63ea9123ee976bac7a3ac8f4eaec3170019e877813b2fa6b4f6067cb53266c6d: Status 404 returned error can't find the container with id 63ea9123ee976bac7a3ac8f4eaec3170019e877813b2fa6b4f6067cb53266c6d
Oct 30 19:55:21.622858 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.622837    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8"]
Oct 30 19:55:21.623926 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:55:21.623898    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7540294_e7e5_42a5_8868_a7e518e7398a.slice/crio-be0bb750999020fd5c77dd9a40c5354f52083a714234258ac70da7870ea77a0e WatchSource:0}: Error finding container be0bb750999020fd5c77dd9a40c5354f52083a714234258ac70da7870ea77a0e: Status 404 returned error can't find the container with id be0bb750999020fd5c77dd9a40c5354f52083a714234258ac70da7870ea77a0e
Oct 30 19:55:21.658857 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.658830    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8" event={"ID":"f7540294-e7e5-42a5-8868-a7e518e7398a","Type":"ContainerStarted","Data":"be0bb750999020fd5c77dd9a40c5354f52083a714234258ac70da7870ea77a0e"}
Oct 30 19:55:21.660698 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.660672    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p" event={"ID":"9af6c27c-16a0-4d18-a328-06b585f944eb","Type":"ContainerStarted","Data":"1a4eeb1e2b44351e97a9080f5cdacaba61953f4d4427e154488ba596fa5b5397"}
Oct 30 19:55:21.660803 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.660722    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p" event={"ID":"9af6c27c-16a0-4d18-a328-06b585f944eb","Type":"ContainerStarted","Data":"63ea9123ee976bac7a3ac8f4eaec3170019e877813b2fa6b4f6067cb53266c6d"}
Oct 30 19:55:21.660929 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.660912    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p"
Oct 30 19:55:21.662495 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.662473    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz" event={"ID":"520ed64e-c5f4-4a6c-b11a-dc5850632383","Type":"ContainerStarted","Data":"2ca0f90803772ecca20fa0176f2dbdcfcba1a84d43109d37e26a114daad0f84c"}
Oct 30 19:55:21.662604 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.662507    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz" event={"ID":"520ed64e-c5f4-4a6c-b11a-dc5850632383","Type":"ContainerStarted","Data":"5c315d095a38e456089b0a57f6a96068ce30725d7b9e3b49fd07521f910147bb"}
Oct 30 19:55:21.662604 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.662597    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"
Oct 30 19:55:21.744644 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.744605    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz" podStartSLOduration=1.46239054 podStartE2EDuration="1.744593383s" totalImagesPullingTime="282.202843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:21.743554125 +0000 UTC m=+2075.042816253" watchObservedRunningTime="2026-10-30 19:55:21.744593383 +0000 UTC m=+2075.043855511"
Oct 30 19:55:21.745236 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.745208    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p" podStartSLOduration=1.745200165 podStartE2EDuration="1.745200165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:21.703179648 +0000 UTC m=+2075.002441776" watchObservedRunningTime="2026-10-30 19:55:21.745200165 +0000 UTC m=+2075.044462295"
Oct 30 19:55:21.966199 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:21.966166    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:22.120818 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:22.120788    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:22.267247 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:22.267160    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:22.676681 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:22.676640    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8" event={"ID":"f7540294-e7e5-42a5-8868-a7e518e7398a","Type":"ContainerStarted","Data":"9a7875266f27a28e1f6a776f171f42d5e5d306c01480ce9315f30c4a86ee5694"}
Oct 30 19:55:22.677071 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:22.676863    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8"
Oct 30 19:55:22.718150 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:22.718097    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8" podStartSLOduration=2.718085589 podStartE2EDuration="2.718085589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:22.717023441 +0000 UTC m=+2076.016285570" watchObservedRunningTime="2026-10-30 19:55:22.718085589 +0000 UTC m=+2076.017347725"
Oct 30 19:55:23.774515 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:23.774477    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:25.180033 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:25.180003    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:26.700824 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:26.700788    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:27.682769 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:27.682736    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"
Oct 30 19:55:28.288176 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:28.288090    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:28.952944 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:28.952889    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:29.783758 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:29.783727    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:29.927714 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:29.927658    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:30.128373 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:30.128343    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:31.573076 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:31.573024    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:31.719221 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:31.719196    4712 generic.go:356] "Generic (PLEG): container finished" podID="4ece469f-27b4-48f2-b640-86d165397d23" containerID="01fe9eb235d635d9864dccd8cab21df03dbc4f898c5f7f65ef8d783387e38477" exitCode=0
Oct 30 19:55:31.719399 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:31.719275    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c" event={"ID":"4ece469f-27b4-48f2-b640-86d165397d23","Type":"ContainerDied","Data":"01fe9eb235d635d9864dccd8cab21df03dbc4f898c5f7f65ef8d783387e38477"}
Oct 30 19:55:31.719756 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:31.719737    4712 scope.go:122] "RemoveContainer" containerID="01fe9eb235d635d9864dccd8cab21df03dbc4f898c5f7f65ef8d783387e38477"
Oct 30 19:55:31.720934 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:31.720905    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:55:31.721017 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:31.720972    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:55:31.721017 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:31.721006    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:55:32.680828 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:32.680791    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p"
Oct 30 19:55:32.740395 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:32.740230    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c" event={"ID":"4ece469f-27b4-48f2-b640-86d165397d23","Type":"ContainerStarted","Data":"b6b207de477b9aa5c51984cf95b345f605cf0a7751b81ec12feb5ed26789f240"}
Oct 30 19:55:32.788534 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:32.788503    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:32.789662 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:32.789638    4712 patch_prober.go:29] interesting pod/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:55:32.789798 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:32.789703    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c" podUID="4ece469f-27b4-48f2-b640-86d165397d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 19:55:33.214888 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.214835    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:33.467092 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.467005    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"]
Oct 30 19:55:33.470694 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.470674    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.485695 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.485670    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-1f3297\"/\"kube-root-ca.crt\""
Oct 30 19:55:33.497837 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.497817    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-1f3297\"/\"openshift-service-ca.crt\""
Oct 30 19:55:33.497992 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.497862    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-1f3297\"/\"default-dockercfg-s5bdj\""
Oct 30 19:55:33.498096 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.498029    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-1f3297\"/\"simulator-single-main-1f3297-kserve-self-signed-certs\""
Oct 30 19:55:33.501817 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.501791    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"]
Oct 30 19:55:33.534175 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.534138    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-model-cache\") pod \"simulator-single-main-1f3297-kserve-796656c47b-q5j9f\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") " pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.534426 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.534406    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-tmp-dir\") pod \"simulator-single-main-1f3297-kserve-796656c47b-q5j9f\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") " pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.534643 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.534618    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-home\") pod \"simulator-single-main-1f3297-kserve-796656c47b-q5j9f\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") " pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.534767 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.534738    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-dshm\") pod \"simulator-single-main-1f3297-kserve-796656c47b-q5j9f\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") " pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.534911 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.534879    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkdjl\" (UniqueName: \"kubernetes.io/projected/07f47e76-6085-43f5-9803-018663f1746b-kube-api-access-kkdjl\") pod \"simulator-single-main-1f3297-kserve-796656c47b-q5j9f\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") " pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.534993 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.534949    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f47e76-6085-43f5-9803-018663f1746b-tls-certs\") pod \"simulator-single-main-1f3297-kserve-796656c47b-q5j9f\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") " pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.595966 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.595942    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:33.635461 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.635432    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-dshm\") pod \"simulator-single-main-1f3297-kserve-796656c47b-q5j9f\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") " pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.635589 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.635492    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkdjl\" (UniqueName: \"kubernetes.io/projected/07f47e76-6085-43f5-9803-018663f1746b-kube-api-access-kkdjl\") pod \"simulator-single-main-1f3297-kserve-796656c47b-q5j9f\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") " pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.635589 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.635517    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f47e76-6085-43f5-9803-018663f1746b-tls-certs\") pod \"simulator-single-main-1f3297-kserve-796656c47b-q5j9f\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") " pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.635589 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.635548    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-model-cache\") pod \"simulator-single-main-1f3297-kserve-796656c47b-q5j9f\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") " pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.635589 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.635563    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-tmp-dir\") pod \"simulator-single-main-1f3297-kserve-796656c47b-q5j9f\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") " pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.635884 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.635689    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-home\") pod \"simulator-single-main-1f3297-kserve-796656c47b-q5j9f\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") " pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.636008 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.635964    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-model-cache\") pod \"simulator-single-main-1f3297-kserve-796656c47b-q5j9f\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") " pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.636071 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.636035    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-tmp-dir\") pod \"simulator-single-main-1f3297-kserve-796656c47b-q5j9f\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") " pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.636117 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.636068    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-home\") pod \"simulator-single-main-1f3297-kserve-796656c47b-q5j9f\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") " pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.637825 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.637798    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-dshm\") pod \"simulator-single-main-1f3297-kserve-796656c47b-q5j9f\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") " pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.637949 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.637931    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f47e76-6085-43f5-9803-018663f1746b-tls-certs\") pod \"simulator-single-main-1f3297-kserve-796656c47b-q5j9f\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") " pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.642910 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.642892    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:33.654634 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.654615    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkdjl\" (UniqueName: \"kubernetes.io/projected/07f47e76-6085-43f5-9803-018663f1746b-kube-api-access-kkdjl\") pod \"simulator-single-main-1f3297-kserve-796656c47b-q5j9f\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") " pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.684196 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.684174    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8"
Oct 30 19:55:33.783090 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.783010    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:33.788457 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.788431    4712 patch_prober.go:29] interesting pod/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:55:33.788591 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:33.788487    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c" podUID="4ece469f-27b4-48f2-b640-86d165397d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 19:55:34.128433 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:34.128205    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"]
Oct 30 19:55:34.134043 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:55:34.134015    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07f47e76_6085_43f5_9803_018663f1746b.slice/crio-023c758d5e28d494da558c2f8af8415ab22e2baa3335a11655d2a4c6475f472c WatchSource:0}: Error finding container 023c758d5e28d494da558c2f8af8415ab22e2baa3335a11655d2a4c6475f472c: Status 404 returned error can't find the container with id 023c758d5e28d494da558c2f8af8415ab22e2baa3335a11655d2a4c6475f472c
Oct 30 19:55:34.221157 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:34.221071    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:34.757318 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:34.757283    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f" event={"ID":"07f47e76-6085-43f5-9803-018663f1746b","Type":"ContainerStarted","Data":"39cc7a28ac8bd13a4f5b57a547dacf51543cb75b641e100fea00209c99483078"}
Oct 30 19:55:34.757858 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:34.757326    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f" event={"ID":"07f47e76-6085-43f5-9803-018663f1746b","Type":"ContainerStarted","Data":"023c758d5e28d494da558c2f8af8415ab22e2baa3335a11655d2a4c6475f472c"}
Oct 30 19:55:34.783154 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:34.783124    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:34.784003 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:34.783971    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:34.788638 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:34.788614    4712 patch_prober.go:29] interesting pod/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:55:34.788782 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:34.788666    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c" podUID="4ece469f-27b4-48f2-b640-86d165397d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 19:55:34.798109 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:34.798065    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f" podStartSLOduration=1.798054154 podStartE2EDuration="1.798054154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:34.797009178 +0000 UTC m=+2088.096271307" watchObservedRunningTime="2026-10-30 19:55:34.798054154 +0000 UTC m=+2088.097316283"
Oct 30 19:55:35.788635 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:35.788601    4712 patch_prober.go:29] interesting pod/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:55:35.789039 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:35.788661    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c" podUID="4ece469f-27b4-48f2-b640-86d165397d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 19:55:36.373415 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:55:36.373351    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:55:36.529920 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:36.529885    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:36.788559 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:36.788471    4712 patch_prober.go:29] interesting pod/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:55:36.788559 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:36.788533    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c" podUID="4ece469f-27b4-48f2-b640-86d165397d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 19:55:38.091722 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:38.091687    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:38.237878 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:38.237850    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:38.240379 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:38.240338    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:55:39.584185 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:39.584152    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:41.087214 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:41.087140    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:42.642154 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:42.642120    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:43.783635 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:43.783607    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:43.797233 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:43.797210    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:43.808734 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:43.808709    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:55:44.192001 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:44.191863    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:45.584417 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:45.584378    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:47.124656 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:47.124631    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:55:47.126232 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:47.126207    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 19:55:47.146197 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:47.146172    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:48.682663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:48.682625    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:50.173954 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:50.173913    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:51.791991 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:51.791955    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:53.289695 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:53.289659    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:54.825573 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:54.825540    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:56.285136 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:56.285099    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:57.772827 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:57.772793    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:55:59.331701 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:55:59.331603    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:00.781776 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:00.781723    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:02.292791 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:02.292745    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:03.824849 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:03.824810    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:05.295017 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:05.294968    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:06.373024 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:56:06.372986    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:56:06.884237 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:06.884205    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:08.357626 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:08.357590    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:09.916336 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:09.916299    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:11.469465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:11.469428    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:12.982424 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:12.982384    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:14.283273 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:14.283246    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:14.510931 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:14.510902    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:16.262771 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:16.262729    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:17.876846 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:17.876814    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:19.191992 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:19.191960    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"]
Oct 30 19:56:19.325270 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:19.325229    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:19.490081 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:19.489994    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:20.977135 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:20.977105    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:22.518892 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:22.518860    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:23.992258 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:23.992227    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:24.851412 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:24.851380    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8"]
Oct 30 19:56:24.885469 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:24.885435    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p"]
Oct 30 19:56:25.294709 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.294678    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"]
Oct 30 19:56:25.295171 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.294893    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz" podUID="520ed64e-c5f4-4a6c-b11a-dc5850632383" containerName="maas-api" containerID="cri-o://2ca0f90803772ecca20fa0176f2dbdcfcba1a84d43109d37e26a114daad0f84c" gracePeriod=30
Oct 30 19:56:25.753595 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.753569    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"
Oct 30 19:56:25.789111 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.789090    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/520ed64e-c5f4-4a6c-b11a-dc5850632383-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/520ed64e-c5f4-4a6c-b11a-dc5850632383-maas-api-tls\") pod \"520ed64e-c5f4-4a6c-b11a-dc5850632383\" (UID: \"520ed64e-c5f4-4a6c-b11a-dc5850632383\") "
Oct 30 19:56:25.789224 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.789156    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/520ed64e-c5f4-4a6c-b11a-dc5850632383-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/520ed64e-c5f4-4a6c-b11a-dc5850632383-openshift-service-ca\") pod \"520ed64e-c5f4-4a6c-b11a-dc5850632383\" (UID: \"520ed64e-c5f4-4a6c-b11a-dc5850632383\") "
Oct 30 19:56:25.789224 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.789193    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/520ed64e-c5f4-4a6c-b11a-dc5850632383-metrics-certs\" (UniqueName: \"kubernetes.io/secret/520ed64e-c5f4-4a6c-b11a-dc5850632383-metrics-certs\") pod \"520ed64e-c5f4-4a6c-b11a-dc5850632383\" (UID: \"520ed64e-c5f4-4a6c-b11a-dc5850632383\") "
Oct 30 19:56:25.789314 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.789223    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520ed64e-c5f4-4a6c-b11a-dc5850632383-kube-api-access-f9vd4\" (UniqueName: \"kubernetes.io/projected/520ed64e-c5f4-4a6c-b11a-dc5850632383-kube-api-access-f9vd4\") pod \"520ed64e-c5f4-4a6c-b11a-dc5850632383\" (UID: \"520ed64e-c5f4-4a6c-b11a-dc5850632383\") "
Oct 30 19:56:25.789577 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.789551    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/520ed64e-c5f4-4a6c-b11a-dc5850632383-openshift-service-ca" pod "520ed64e-c5f4-4a6c-b11a-dc5850632383" (UID: "520ed64e-c5f4-4a6c-b11a-dc5850632383"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:25.791318 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.791278    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/520ed64e-c5f4-4a6c-b11a-dc5850632383-maas-api-tls" pod "520ed64e-c5f4-4a6c-b11a-dc5850632383" (UID: "520ed64e-c5f4-4a6c-b11a-dc5850632383"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:25.791483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.791396    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/520ed64e-c5f4-4a6c-b11a-dc5850632383-metrics-certs" pod "520ed64e-c5f4-4a6c-b11a-dc5850632383" (UID: "520ed64e-c5f4-4a6c-b11a-dc5850632383"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:25.791483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.791399    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520ed64e-c5f4-4a6c-b11a-dc5850632383-kube-api-access-f9vd4" pod "520ed64e-c5f4-4a6c-b11a-dc5850632383" (UID: "520ed64e-c5f4-4a6c-b11a-dc5850632383"). InnerVolumeSpecName "kube-api-access-f9vd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:25.890777 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.890748    4712 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/520ed64e-c5f4-4a6c-b11a-dc5850632383-metrics-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:25.890777 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.890774    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9vd4\" (UniqueName: \"kubernetes.io/projected/520ed64e-c5f4-4a6c-b11a-dc5850632383-kube-api-access-f9vd4\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:25.890923 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.890784    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/520ed64e-c5f4-4a6c-b11a-dc5850632383-maas-api-tls\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:25.890923 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.890793    4712 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/520ed64e-c5f4-4a6c-b11a-dc5850632383-openshift-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:25.975606 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.975581    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:25.986487 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.986458    4712 generic.go:356] "Generic (PLEG): container finished" podID="520ed64e-c5f4-4a6c-b11a-dc5850632383" containerID="2ca0f90803772ecca20fa0176f2dbdcfcba1a84d43109d37e26a114daad0f84c" exitCode=0
Oct 30 19:56:25.986590 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.986502    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"
Oct 30 19:56:25.986590 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.986545    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz" event={"ID":"520ed64e-c5f4-4a6c-b11a-dc5850632383","Type":"ContainerDied","Data":"2ca0f90803772ecca20fa0176f2dbdcfcba1a84d43109d37e26a114daad0f84c"}
Oct 30 19:56:25.986670 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.986601    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz" event={"ID":"520ed64e-c5f4-4a6c-b11a-dc5850632383","Type":"ContainerDied","Data":"5c315d095a38e456089b0a57f6a96068ce30725d7b9e3b49fd07521f910147bb"}
Oct 30 19:56:25.986670 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.986624    4712 scope.go:122] "RemoveContainer" containerID="2ca0f90803772ecca20fa0176f2dbdcfcba1a84d43109d37e26a114daad0f84c"
Oct 30 19:56:25.996082 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.996066    4712 scope.go:122] "RemoveContainer" containerID="2ca0f90803772ecca20fa0176f2dbdcfcba1a84d43109d37e26a114daad0f84c"
Oct 30 19:56:25.996334 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:56:25.996317    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca0f90803772ecca20fa0176f2dbdcfcba1a84d43109d37e26a114daad0f84c\": container with ID starting with 2ca0f90803772ecca20fa0176f2dbdcfcba1a84d43109d37e26a114daad0f84c not found: ID does not exist" containerID="2ca0f90803772ecca20fa0176f2dbdcfcba1a84d43109d37e26a114daad0f84c"
Oct 30 19:56:25.996445 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:25.996340    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca0f90803772ecca20fa0176f2dbdcfcba1a84d43109d37e26a114daad0f84c"} err="failed to get container status \"2ca0f90803772ecca20fa0176f2dbdcfcba1a84d43109d37e26a114daad0f84c\": rpc error: code = NotFound desc = could not find container \"2ca0f90803772ecca20fa0176f2dbdcfcba1a84d43109d37e26a114daad0f84c\": container with ID starting with 2ca0f90803772ecca20fa0176f2dbdcfcba1a84d43109d37e26a114daad0f84c not found: ID does not exist"
Oct 30 19:56:26.036197 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:26.036175    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"]
Oct 30 19:56:26.042905 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:26.042883    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1f3297-56d97fc7b4-qkmkz"]
Oct 30 19:56:26.856653 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:26.856610    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="520ed64e-c5f4-4a6c-b11a-dc5850632383" path="/var/lib/kubelet/pods/520ed64e-c5f4-4a6c-b11a-dc5850632383/volumes"
Oct 30 19:56:27.464629 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:27.464596    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:27.609488 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:27.609453    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:28.078616 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:28.078580    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:29.852395 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:29.852340    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8" podUID="f7540294-e7e5-42a5-8868-a7e518e7398a" containerName="payload-processing" containerID="cri-o://9a7875266f27a28e1f6a776f171f42d5e5d306c01480ce9315f30c4a86ee5694" gracePeriod=25
Oct 30 19:56:29.886662 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:29.886618    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p" podUID="9af6c27c-16a0-4d18-a328-06b585f944eb" containerName="payload-pre-processing" containerID="cri-o://1a4eeb1e2b44351e97a9080f5cdacaba61953f4d4427e154488ba596fa5b5397" gracePeriod=25
Oct 30 19:56:32.678328 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:32.678292    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p" podUID="9af6c27c-16a0-4d18-a328-06b585f944eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:56:33.682784 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:33.682742    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8" podUID="f7540294-e7e5-42a5-8868-a7e518e7398a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:56:33.926015 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:33.925984    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:34.072610 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.072510    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:34.210639 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.210608    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"]
Oct 30 19:56:34.211151 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.211110    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c" podUID="4ece469f-27b4-48f2-b640-86d165397d23" containerName="istio-proxy" containerID="cri-o://b6b207de477b9aa5c51984cf95b345f605cf0a7751b81ec12feb5ed26789f240" gracePeriod=30
Oct 30 19:56:34.213015 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.212983    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f" podUID="07f47e76-6085-43f5-9803-018663f1746b" containerName="main" containerID="cri-o://39cc7a28ac8bd13a4f5b57a547dacf51543cb75b641e100fea00209c99483078" gracePeriod=45
Oct 30 19:56:34.286742 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.286713    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:34.431883 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.431854    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:34.462955 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.462926    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:56:34.564259 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.564216    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07f47e76-6085-43f5-9803-018663f1746b-tls-certs\" (UniqueName: \"kubernetes.io/secret/07f47e76-6085-43f5-9803-018663f1746b-tls-certs\") pod \"07f47e76-6085-43f5-9803-018663f1746b\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") "
Oct 30 19:56:34.564449 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.564351    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-home\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-home\") pod \"07f47e76-6085-43f5-9803-018663f1746b\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") "
Oct 30 19:56:34.564501 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.564473    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-dshm\") pod \"07f47e76-6085-43f5-9803-018663f1746b\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") "
Oct 30 19:56:34.564563 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.564545    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-model-cache\") pod \"07f47e76-6085-43f5-9803-018663f1746b\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") "
Oct 30 19:56:34.564625 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.564580    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-home" pod "07f47e76-6085-43f5-9803-018663f1746b" (UID: "07f47e76-6085-43f5-9803-018663f1746b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:34.564625 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.564589    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f47e76-6085-43f5-9803-018663f1746b-kube-api-access-kkdjl\" (UniqueName: \"kubernetes.io/projected/07f47e76-6085-43f5-9803-018663f1746b-kube-api-access-kkdjl\") pod \"07f47e76-6085-43f5-9803-018663f1746b\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") "
Oct 30 19:56:34.564746 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.564689    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-tmp-dir\") pod \"07f47e76-6085-43f5-9803-018663f1746b\" (UID: \"07f47e76-6085-43f5-9803-018663f1746b\") "
Oct 30 19:56:34.565150 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.564826    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-model-cache" pod "07f47e76-6085-43f5-9803-018663f1746b" (UID: "07f47e76-6085-43f5-9803-018663f1746b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:34.565150 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.565008    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-tmp-dir" pod "07f47e76-6085-43f5-9803-018663f1746b" (UID: "07f47e76-6085-43f5-9803-018663f1746b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:34.565150 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.565088    4712 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-model-cache\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:34.565150 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.565124    4712 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-tmp-dir\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:34.565150 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.565132    4712 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-home\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:34.567043 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.567013    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f47e76-6085-43f5-9803-018663f1746b-tls-certs" pod "07f47e76-6085-43f5-9803-018663f1746b" (UID: "07f47e76-6085-43f5-9803-018663f1746b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:34.567145 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.567072    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-dshm" pod "07f47e76-6085-43f5-9803-018663f1746b" (UID: "07f47e76-6085-43f5-9803-018663f1746b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:34.567210 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.567187    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f47e76-6085-43f5-9803-018663f1746b-kube-api-access-kkdjl" pod "07f47e76-6085-43f5-9803-018663f1746b" (UID: "07f47e76-6085-43f5-9803-018663f1746b"). InnerVolumeSpecName "kube-api-access-kkdjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:34.667106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.667068    4712 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07f47e76-6085-43f5-9803-018663f1746b-dshm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:34.667106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.667098    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkdjl\" (UniqueName: \"kubernetes.io/projected/07f47e76-6085-43f5-9803-018663f1746b-kube-api-access-kkdjl\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:34.667106 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:34.667109    4712 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f47e76-6085-43f5-9803-018663f1746b-tls-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:35.027545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:35.027456    4712 generic.go:356] "Generic (PLEG): container finished" podID="07f47e76-6085-43f5-9803-018663f1746b" containerID="39cc7a28ac8bd13a4f5b57a547dacf51543cb75b641e100fea00209c99483078" exitCode=0
Oct 30 19:56:35.027545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:35.027501    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"
Oct 30 19:56:35.027545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:35.027509    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f" event={"ID":"07f47e76-6085-43f5-9803-018663f1746b","Type":"ContainerDied","Data":"39cc7a28ac8bd13a4f5b57a547dacf51543cb75b641e100fea00209c99483078"}
Oct 30 19:56:35.027545 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:35.027540    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f" event={"ID":"07f47e76-6085-43f5-9803-018663f1746b","Type":"ContainerDied","Data":"023c758d5e28d494da558c2f8af8415ab22e2baa3335a11655d2a4c6475f472c"}
Oct 30 19:56:35.028151 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:35.027555    4712 scope.go:122] "RemoveContainer" containerID="39cc7a28ac8bd13a4f5b57a547dacf51543cb75b641e100fea00209c99483078"
Oct 30 19:56:35.035638 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:35.035620    4712 scope.go:122] "RemoveContainer" containerID="39cc7a28ac8bd13a4f5b57a547dacf51543cb75b641e100fea00209c99483078"
Oct 30 19:56:35.035933 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:56:35.035913    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39cc7a28ac8bd13a4f5b57a547dacf51543cb75b641e100fea00209c99483078\": container with ID starting with 39cc7a28ac8bd13a4f5b57a547dacf51543cb75b641e100fea00209c99483078 not found: ID does not exist" containerID="39cc7a28ac8bd13a4f5b57a547dacf51543cb75b641e100fea00209c99483078"
Oct 30 19:56:35.036004 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:35.035945    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39cc7a28ac8bd13a4f5b57a547dacf51543cb75b641e100fea00209c99483078"} err="failed to get container status \"39cc7a28ac8bd13a4f5b57a547dacf51543cb75b641e100fea00209c99483078\": rpc error: code = NotFound desc = could not find container \"39cc7a28ac8bd13a4f5b57a547dacf51543cb75b641e100fea00209c99483078\": container with ID starting with 39cc7a28ac8bd13a4f5b57a547dacf51543cb75b641e100fea00209c99483078 not found: ID does not exist"
Oct 30 19:56:35.079424 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:35.079395    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"]
Oct 30 19:56:35.083218 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:35.083197    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-1f3297/simulator-single-main-1f3297-kserve-796656c47b-q5j9f"]
Oct 30 19:56:36.371913 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:56:36.371875    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:56:36.853636 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:36.853555    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f47e76-6085-43f5-9803-018663f1746b" path="/var/lib/kubelet/pods/07f47e76-6085-43f5-9803-018663f1746b/volumes"
Oct 30 19:56:38.240747 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:38.240708    4712 patch_prober.go:29] interesting pod/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:56:38.241240 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:38.240782    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c" podUID="4ece469f-27b4-48f2-b640-86d165397d23" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:56:39.458265 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.458242    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:56:39.508087 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.508033    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-workload-certs\") pod \"4ece469f-27b4-48f2-b640-86d165397d23\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") "
Oct 30 19:56:39.508087 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.508068    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-istio-envoy\") pod \"4ece469f-27b4-48f2-b640-86d165397d23\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") "
Oct 30 19:56:39.508261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.508104    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-credential-socket\") pod \"4ece469f-27b4-48f2-b640-86d165397d23\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") "
Oct 30 19:56:39.508261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.508128    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-istio-data\") pod \"4ece469f-27b4-48f2-b640-86d165397d23\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") "
Oct 30 19:56:39.508261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.508159    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ece469f-27b4-48f2-b640-86d165397d23-istio-token\" (UniqueName: \"kubernetes.io/projected/4ece469f-27b4-48f2-b640-86d165397d23-istio-token\") pod \"4ece469f-27b4-48f2-b640-86d165397d23\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") "
Oct 30 19:56:39.508261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.508183    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ece469f-27b4-48f2-b640-86d165397d23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ece469f-27b4-48f2-b640-86d165397d23-istiod-ca-cert\") pod \"4ece469f-27b4-48f2-b640-86d165397d23\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") "
Oct 30 19:56:39.508261 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.508247    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-workload-socket\") pod \"4ece469f-27b4-48f2-b640-86d165397d23\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") "
Oct 30 19:56:39.508574 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.508309    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ece469f-27b4-48f2-b640-86d165397d23-kube-api-access-ff2wn\" (UniqueName: \"kubernetes.io/projected/4ece469f-27b4-48f2-b640-86d165397d23-kube-api-access-ff2wn\") pod \"4ece469f-27b4-48f2-b640-86d165397d23\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") "
Oct 30 19:56:39.508574 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.508310    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-workload-certs" pod "4ece469f-27b4-48f2-b640-86d165397d23" (UID: "4ece469f-27b4-48f2-b640-86d165397d23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:39.508574 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.508376    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ece469f-27b4-48f2-b640-86d165397d23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ece469f-27b4-48f2-b640-86d165397d23-istio-podinfo\") pod \"4ece469f-27b4-48f2-b640-86d165397d23\" (UID: \"4ece469f-27b4-48f2-b640-86d165397d23\") "
Oct 30 19:56:39.508574 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.508441    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-credential-socket" pod "4ece469f-27b4-48f2-b640-86d165397d23" (UID: "4ece469f-27b4-48f2-b640-86d165397d23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:39.508574 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.508528    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-istio-data" pod "4ece469f-27b4-48f2-b640-86d165397d23" (UID: "4ece469f-27b4-48f2-b640-86d165397d23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:39.508843 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.508692    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-workload-socket" pod "4ece469f-27b4-48f2-b640-86d165397d23" (UID: "4ece469f-27b4-48f2-b640-86d165397d23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:39.508843 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.508741    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:39.508843 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.508760    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:39.508843 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.508777    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:39.508843 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.508787    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ece469f-27b4-48f2-b640-86d165397d23-istiod-ca-cert" pod "4ece469f-27b4-48f2-b640-86d165397d23" (UID: "4ece469f-27b4-48f2-b640-86d165397d23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:39.510609 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.510586    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ece469f-27b4-48f2-b640-86d165397d23-istio-token" pod "4ece469f-27b4-48f2-b640-86d165397d23" (UID: "4ece469f-27b4-48f2-b640-86d165397d23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:39.510720 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.510599    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ece469f-27b4-48f2-b640-86d165397d23-kube-api-access-ff2wn" pod "4ece469f-27b4-48f2-b640-86d165397d23" (UID: "4ece469f-27b4-48f2-b640-86d165397d23"). InnerVolumeSpecName "kube-api-access-ff2wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:39.511374 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.511332    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-istio-envoy" pod "4ece469f-27b4-48f2-b640-86d165397d23" (UID: "4ece469f-27b4-48f2-b640-86d165397d23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:39.511472 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.511337    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ece469f-27b4-48f2-b640-86d165397d23-istio-podinfo" pod "4ece469f-27b4-48f2-b640-86d165397d23" (UID: "4ece469f-27b4-48f2-b640-86d165397d23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:56:39.609751 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.609719    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ece469f-27b4-48f2-b640-86d165397d23-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:39.609751 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.609747    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ece469f-27b4-48f2-b640-86d165397d23-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:39.609924 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.609760    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:39.609924 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.609773    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff2wn\" (UniqueName: \"kubernetes.io/projected/4ece469f-27b4-48f2-b640-86d165397d23-kube-api-access-ff2wn\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:39.609924 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.609787    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ece469f-27b4-48f2-b640-86d165397d23-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:39.609924 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:39.609800    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ece469f-27b4-48f2-b640-86d165397d23-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:40.053877 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:40.053839    4712 generic.go:356] "Generic (PLEG): container finished" podID="4ece469f-27b4-48f2-b640-86d165397d23" containerID="b6b207de477b9aa5c51984cf95b345f605cf0a7751b81ec12feb5ed26789f240" exitCode=0
Oct 30 19:56:40.054053 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:40.053921    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c" event={"ID":"4ece469f-27b4-48f2-b640-86d165397d23","Type":"ContainerDied","Data":"b6b207de477b9aa5c51984cf95b345f605cf0a7751b81ec12feb5ed26789f240"}
Oct 30 19:56:40.054053 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:40.053976    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c" event={"ID":"4ece469f-27b4-48f2-b640-86d165397d23","Type":"ContainerDied","Data":"52ab1abb37c700352547d979bb763611f1f1e97b5542182f06b54459a145af9b"}
Oct 30 19:56:40.054053 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:40.053997    4712 scope.go:122] "RemoveContainer" containerID="b6b207de477b9aa5c51984cf95b345f605cf0a7751b81ec12feb5ed26789f240"
Oct 30 19:56:40.054053 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:40.053933    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"
Oct 30 19:56:40.062715 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:40.062699    4712 scope.go:122] "RemoveContainer" containerID="01fe9eb235d635d9864dccd8cab21df03dbc4f898c5f7f65ef8d783387e38477"
Oct 30 19:56:40.070227 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:40.070208    4712 scope.go:122] "RemoveContainer" containerID="b6b207de477b9aa5c51984cf95b345f605cf0a7751b81ec12feb5ed26789f240"
Oct 30 19:56:40.070475 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:56:40.070458    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b207de477b9aa5c51984cf95b345f605cf0a7751b81ec12feb5ed26789f240\": container with ID starting with b6b207de477b9aa5c51984cf95b345f605cf0a7751b81ec12feb5ed26789f240 not found: ID does not exist" containerID="b6b207de477b9aa5c51984cf95b345f605cf0a7751b81ec12feb5ed26789f240"
Oct 30 19:56:40.070534 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:40.070483    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b207de477b9aa5c51984cf95b345f605cf0a7751b81ec12feb5ed26789f240"} err="failed to get container status \"b6b207de477b9aa5c51984cf95b345f605cf0a7751b81ec12feb5ed26789f240\": rpc error: code = NotFound desc = could not find container \"b6b207de477b9aa5c51984cf95b345f605cf0a7751b81ec12feb5ed26789f240\": container with ID starting with b6b207de477b9aa5c51984cf95b345f605cf0a7751b81ec12feb5ed26789f240 not found: ID does not exist"
Oct 30 19:56:40.070534 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:40.070497    4712 scope.go:122] "RemoveContainer" containerID="01fe9eb235d635d9864dccd8cab21df03dbc4f898c5f7f65ef8d783387e38477"
Oct 30 19:56:40.070717 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:56:40.070699    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01fe9eb235d635d9864dccd8cab21df03dbc4f898c5f7f65ef8d783387e38477\": container with ID starting with 01fe9eb235d635d9864dccd8cab21df03dbc4f898c5f7f65ef8d783387e38477 not found: ID does not exist" containerID="01fe9eb235d635d9864dccd8cab21df03dbc4f898c5f7f65ef8d783387e38477"
Oct 30 19:56:40.070755 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:40.070721    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01fe9eb235d635d9864dccd8cab21df03dbc4f898c5f7f65ef8d783387e38477"} err="failed to get container status \"01fe9eb235d635d9864dccd8cab21df03dbc4f898c5f7f65ef8d783387e38477\": rpc error: code = NotFound desc = could not find container \"01fe9eb235d635d9864dccd8cab21df03dbc4f898c5f7f65ef8d783387e38477\": container with ID starting with 01fe9eb235d635d9864dccd8cab21df03dbc4f898c5f7f65ef8d783387e38477 not found: ID does not exist"
Oct 30 19:56:40.105737 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:40.105712    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"]
Oct 30 19:56:40.117756 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:40.117727    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1f3297-openshift-default-56bdc48cd9d7b7c"]
Oct 30 19:56:40.850436 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:40.850404    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ece469f-27b4-48f2-b640-86d165397d23" path="/var/lib/kubelet/pods/4ece469f-27b4-48f2-b640-86d165397d23/volumes"
Oct 30 19:56:42.678878 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:42.678844    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p" podUID="9af6c27c-16a0-4d18-a328-06b585f944eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:56:43.682680 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:43.682646    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8" podUID="f7540294-e7e5-42a5-8868-a7e518e7398a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:56:50.105577 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:50.105538    4712 generic.go:356] "Generic (PLEG): container finished" podID="9af6c27c-16a0-4d18-a328-06b585f944eb" containerID="1a4eeb1e2b44351e97a9080f5cdacaba61953f4d4427e154488ba596fa5b5397" exitCode=0
Oct 30 19:56:50.106093 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:50.105637    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p" event={"ID":"9af6c27c-16a0-4d18-a328-06b585f944eb","Type":"ContainerDied","Data":"1a4eeb1e2b44351e97a9080f5cdacaba61953f4d4427e154488ba596fa5b5397"}
Oct 30 19:56:50.107734 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:50.107648    4712 generic.go:356] "Generic (PLEG): container finished" podID="f7540294-e7e5-42a5-8868-a7e518e7398a" containerID="9a7875266f27a28e1f6a776f171f42d5e5d306c01480ce9315f30c4a86ee5694" exitCode=0
Oct 30 19:56:50.107873 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:50.107733    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8" event={"ID":"f7540294-e7e5-42a5-8868-a7e518e7398a","Type":"ContainerDied","Data":"9a7875266f27a28e1f6a776f171f42d5e5d306c01480ce9315f30c4a86ee5694"}
Oct 30 19:56:50.161321 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:50.161290    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p"
Oct 30 19:56:50.182669 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:50.179528    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8"
Oct 30 19:56:50.293159 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:50.293126    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9af6c27c-16a0-4d18-a328-06b585f944eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af6c27c-16a0-4d18-a328-06b585f944eb-plugins-config-volume\") pod \"9af6c27c-16a0-4d18-a328-06b585f944eb\" (UID: \"9af6c27c-16a0-4d18-a328-06b585f944eb\") "
Oct 30 19:56:50.293323 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:50.293185    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7540294-e7e5-42a5-8868-a7e518e7398a-kube-api-access-hdfzz\" (UniqueName: \"kubernetes.io/projected/f7540294-e7e5-42a5-8868-a7e518e7398a-kube-api-access-hdfzz\") pod \"f7540294-e7e5-42a5-8868-a7e518e7398a\" (UID: \"f7540294-e7e5-42a5-8868-a7e518e7398a\") "
Oct 30 19:56:50.293323 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:50.293208    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7540294-e7e5-42a5-8868-a7e518e7398a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7540294-e7e5-42a5-8868-a7e518e7398a-plugins-config-volume\") pod \"f7540294-e7e5-42a5-8868-a7e518e7398a\" (UID: \"f7540294-e7e5-42a5-8868-a7e518e7398a\") "
Oct 30 19:56:50.293434 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:50.293321    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af6c27c-16a0-4d18-a328-06b585f944eb-kube-api-access-rfrmk\" (UniqueName: \"kubernetes.io/projected/9af6c27c-16a0-4d18-a328-06b585f944eb-kube-api-access-rfrmk\") pod \"9af6c27c-16a0-4d18-a328-06b585f944eb\" (UID: \"9af6c27c-16a0-4d18-a328-06b585f944eb\") "
Oct 30 19:56:50.293622 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:50.293591    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af6c27c-16a0-4d18-a328-06b585f944eb-plugins-config-volume" pod "9af6c27c-16a0-4d18-a328-06b585f944eb" (UID: "9af6c27c-16a0-4d18-a328-06b585f944eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:50.293622 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:50.293595    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7540294-e7e5-42a5-8868-a7e518e7398a-plugins-config-volume" pod "f7540294-e7e5-42a5-8868-a7e518e7398a" (UID: "f7540294-e7e5-42a5-8868-a7e518e7398a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:50.295509 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:50.295486    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af6c27c-16a0-4d18-a328-06b585f944eb-kube-api-access-rfrmk" pod "9af6c27c-16a0-4d18-a328-06b585f944eb" (UID: "9af6c27c-16a0-4d18-a328-06b585f944eb"). InnerVolumeSpecName "kube-api-access-rfrmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:50.295591 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:50.295518    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7540294-e7e5-42a5-8868-a7e518e7398a-kube-api-access-hdfzz" pod "f7540294-e7e5-42a5-8868-a7e518e7398a" (UID: "f7540294-e7e5-42a5-8868-a7e518e7398a"). InnerVolumeSpecName "kube-api-access-hdfzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:50.394289 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:50.394250    4712 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af6c27c-16a0-4d18-a328-06b585f944eb-plugins-config-volume\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:50.394289 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:50.394290    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdfzz\" (UniqueName: \"kubernetes.io/projected/f7540294-e7e5-42a5-8868-a7e518e7398a-kube-api-access-hdfzz\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:50.394559 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:50.394304    4712 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7540294-e7e5-42a5-8868-a7e518e7398a-plugins-config-volume\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:50.394559 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:50.394318    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfrmk\" (UniqueName: \"kubernetes.io/projected/9af6c27c-16a0-4d18-a328-06b585f944eb-kube-api-access-rfrmk\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 19:56:51.113752 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:51.113673    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p"
Oct 30 19:56:51.113752 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:51.113686    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p" event={"ID":"9af6c27c-16a0-4d18-a328-06b585f944eb","Type":"ContainerDied","Data":"63ea9123ee976bac7a3ac8f4eaec3170019e877813b2fa6b4f6067cb53266c6d"}
Oct 30 19:56:51.114250 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:51.113776    4712 scope.go:122] "RemoveContainer" containerID="1a4eeb1e2b44351e97a9080f5cdacaba61953f4d4427e154488ba596fa5b5397"
Oct 30 19:56:51.115245 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:51.115232    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8"
Oct 30 19:56:51.115245 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:51.115236    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8" event={"ID":"f7540294-e7e5-42a5-8868-a7e518e7398a","Type":"ContainerDied","Data":"be0bb750999020fd5c77dd9a40c5354f52083a714234258ac70da7870ea77a0e"}
Oct 30 19:56:51.122330 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:51.122310    4712 scope.go:122] "RemoveContainer" containerID="9a7875266f27a28e1f6a776f171f42d5e5d306c01480ce9315f30c4a86ee5694"
Oct 30 19:56:51.168254 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:51.168234    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p"]
Oct 30 19:56:51.180190 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:51.180169    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1f3297-6d65d4rw2p"]
Oct 30 19:56:51.217528 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:51.217508    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8"]
Oct 30 19:56:51.227056 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:51.227037    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1f3297-6b89dff9czdwc8"]
Oct 30 19:56:52.850615 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:52.850581    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af6c27c-16a0-4d18-a328-06b585f944eb" path="/var/lib/kubelet/pods/9af6c27c-16a0-4d18-a328-06b585f944eb/volumes"
Oct 30 19:56:52.851004 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:52.850924    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7540294-e7e5-42a5-8868-a7e518e7398a" path="/var/lib/kubelet/pods/f7540294-e7e5-42a5-8868-a7e518e7398a/volumes"
Oct 30 19:56:56.746423 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:56.746390    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:57.469730 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:57.469512    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:56:57.657104 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:56:57.657072    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:03.252013 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:57:03.251981    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:06.372230 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:57:06.372192    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:57:34.440821 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:57:34.440743    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:57:36.372045 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:57:36.372009    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:58:06.372731 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:58:06.372692    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:58:34.632795 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:34.632760    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:36.372737 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:58:36.372701    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:58:51.669767 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:51.669737    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:52.041236 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041155    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"]
Oct 30 19:58:52.041619 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041601    4712 cpu_manager.go:414] "containerMap: removing container" podUID="4ece469f-27b4-48f2-b640-86d165397d23" containerName="istio-proxy"
Oct 30 19:58:52.041728 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041620    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ece469f-27b4-48f2-b640-86d165397d23" containerName="istio-proxy"
Oct 30 19:58:52.041728 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041645    4712 cpu_manager.go:414] "containerMap: removing container" podUID="4ece469f-27b4-48f2-b640-86d165397d23" containerName="istio-proxy"
Oct 30 19:58:52.041728 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041653    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ece469f-27b4-48f2-b640-86d165397d23" containerName="istio-proxy"
Oct 30 19:58:52.041728 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041668    4712 cpu_manager.go:414] "containerMap: removing container" podUID="f7540294-e7e5-42a5-8868-a7e518e7398a" containerName="payload-processing"
Oct 30 19:58:52.041728 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041676    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7540294-e7e5-42a5-8868-a7e518e7398a" containerName="payload-processing"
Oct 30 19:58:52.041728 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041690    4712 cpu_manager.go:414] "containerMap: removing container" podUID="07f47e76-6085-43f5-9803-018663f1746b" containerName="main"
Oct 30 19:58:52.041728 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041697    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f47e76-6085-43f5-9803-018663f1746b" containerName="main"
Oct 30 19:58:52.042082 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041733    4712 cpu_manager.go:414] "containerMap: removing container" podUID="520ed64e-c5f4-4a6c-b11a-dc5850632383" containerName="maas-api"
Oct 30 19:58:52.042082 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041741    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520ed64e-c5f4-4a6c-b11a-dc5850632383" containerName="maas-api"
Oct 30 19:58:52.042082 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041754    4712 cpu_manager.go:414] "containerMap: removing container" podUID="9af6c27c-16a0-4d18-a328-06b585f944eb" containerName="payload-pre-processing"
Oct 30 19:58:52.042082 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041762    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af6c27c-16a0-4d18-a328-06b585f944eb" containerName="payload-pre-processing"
Oct 30 19:58:52.042082 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041857    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ece469f-27b4-48f2-b640-86d165397d23" containerName="istio-proxy"
Oct 30 19:58:52.042082 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041873    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ece469f-27b4-48f2-b640-86d165397d23" containerName="istio-proxy"
Oct 30 19:58:52.042082 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041884    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7540294-e7e5-42a5-8868-a7e518e7398a" containerName="payload-processing"
Oct 30 19:58:52.042082 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041893    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f47e76-6085-43f5-9803-018663f1746b" containerName="main"
Oct 30 19:58:52.042082 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041906    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="520ed64e-c5f4-4a6c-b11a-dc5850632383" containerName="maas-api"
Oct 30 19:58:52.042082 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.041918    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af6c27c-16a0-4d18-a328-06b585f944eb" containerName="payload-pre-processing"
Oct 30 19:58:52.045043 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.045023    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.062497 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.062476    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-a25f47-openshift-default-dockercfg-lxrb8\""
Oct 30 19:58:52.076798 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.076772    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"]
Oct 30 19:58:52.103827 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.103803    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-token\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.103929 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.103834    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-envoy\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.103929 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.103858    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-podinfo\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.104001 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.103925    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n24vw\" (UniqueName: \"kubernetes.io/projected/0171ca81-b7fc-41ee-bee3-2cec16b84f48-kube-api-access-n24vw\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.104048 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.104027    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istiod-ca-cert\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.104101 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.104086    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-data\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.104140 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.104109    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-workload-socket\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.104195 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.104181    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-credential-socket\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.104247 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.104235    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-workload-certs\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.137231 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.137202    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:52.205448 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.205413    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-workload-certs\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.205613 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.205466    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-token\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.205613 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.205498    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-envoy\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.205613 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.205535    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-podinfo\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.205613 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.205567    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n24vw\" (UniqueName: \"kubernetes.io/projected/0171ca81-b7fc-41ee-bee3-2cec16b84f48-kube-api-access-n24vw\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.205613 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.205612    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istiod-ca-cert\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.205918 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.205639    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-data\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.205918 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.205665    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-workload-socket\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.205918 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.205751    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-credential-socket\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.205918 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.205841    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-workload-certs\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.206180 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.206118    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-credential-socket\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.206826 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.206802    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istiod-ca-cert\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.207138 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.207115    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-data\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.207461 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.207436    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-workload-socket\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.209453 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.209427    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-podinfo\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.209578 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.209557    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-envoy\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.227155 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.227127    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-token\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.227842 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.227821    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n24vw\" (UniqueName: \"kubernetes.io/projected/0171ca81-b7fc-41ee-bee3-2cec16b84f48-kube-api-access-n24vw\") pod \"e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") " pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.283446 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.283418    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:52.359129 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.359074    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:52.663635 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.663613    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"]
Oct 30 19:58:52.664533 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:58:52.664509    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0171ca81_b7fc_41ee_bee3_2cec16b84f48.slice/crio-9cdeb6df97985a6dbae7c778c1a5fd5e6abb18588d3d8dae8a4dac2dbfc7e7f7 WatchSource:0}: Error finding container 9cdeb6df97985a6dbae7c778c1a5fd5e6abb18588d3d8dae8a4dac2dbfc7e7f7: Status 404 returned error can't find the container with id 9cdeb6df97985a6dbae7c778c1a5fd5e6abb18588d3d8dae8a4dac2dbfc7e7f7
Oct 30 19:58:52.666497 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.666464    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:58:52.666566 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.666528    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:58:52.666566 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:52.666556    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:58:53.644861 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:53.644822    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh" event={"ID":"0171ca81-b7fc-41ee-bee3-2cec16b84f48","Type":"ContainerStarted","Data":"600071442fb4329ec6a13a45590ac7ea3d6a24cdf41950446dc164ff4decbab9"}
Oct 30 19:58:53.645378 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:53.644881    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh" event={"ID":"0171ca81-b7fc-41ee-bee3-2cec16b84f48","Type":"ContainerStarted","Data":"9cdeb6df97985a6dbae7c778c1a5fd5e6abb18588d3d8dae8a4dac2dbfc7e7f7"}
Oct 30 19:58:54.360150 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:54.360114    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:54.365026 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:54.365001    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:54.421060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:54.421013    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh" podStartSLOduration=2.420999595 podStartE2EDuration="2.420999595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:53.727942642 +0000 UTC m=+2287.027204771" watchObservedRunningTime="2026-10-30 19:58:54.420999595 +0000 UTC m=+2287.720261723"
Oct 30 19:58:54.649648 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:54.649619    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:54.650544 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:54.650525    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:58:56.218978 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:56.218947    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:56.388269 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:56.388236    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:57.434948 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.434911    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"]
Oct 30 19:58:57.438468 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.438450    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"
Oct 30 19:58:57.463910 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.463882    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-a25f47-serving-cert\""
Oct 30 19:58:57.470225 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.470192    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"]
Oct 30 19:58:57.555471 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.555440    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18124ff-856e-437b-875b-897009a6518a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz\" (UID: \"e18124ff-856e-437b-875b-897009a6518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"
Oct 30 19:58:57.555625 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.555486    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18124ff-856e-437b-875b-897009a6518a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz\" (UID: \"e18124ff-856e-437b-875b-897009a6518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"
Oct 30 19:58:57.555625 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.555514    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w75wq\" (UniqueName: \"kubernetes.io/projected/e18124ff-856e-437b-875b-897009a6518a-kube-api-access-w75wq\") pod \"maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz\" (UID: \"e18124ff-856e-437b-875b-897009a6518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"
Oct 30 19:58:57.555697 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.555647    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18124ff-856e-437b-875b-897009a6518a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz\" (UID: \"e18124ff-856e-437b-875b-897009a6518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"
Oct 30 19:58:57.569190 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.569152    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:57.656740 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.656715    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18124ff-856e-437b-875b-897009a6518a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz\" (UID: \"e18124ff-856e-437b-875b-897009a6518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"
Oct 30 19:58:57.656863 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.656775    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18124ff-856e-437b-875b-897009a6518a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz\" (UID: \"e18124ff-856e-437b-875b-897009a6518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"
Oct 30 19:58:57.656863 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.656810    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18124ff-856e-437b-875b-897009a6518a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz\" (UID: \"e18124ff-856e-437b-875b-897009a6518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"
Oct 30 19:58:57.656863 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.656834    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w75wq\" (UniqueName: \"kubernetes.io/projected/e18124ff-856e-437b-875b-897009a6518a-kube-api-access-w75wq\") pod \"maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz\" (UID: \"e18124ff-856e-437b-875b-897009a6518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"
Oct 30 19:58:57.657485 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.657462    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18124ff-856e-437b-875b-897009a6518a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz\" (UID: \"e18124ff-856e-437b-875b-897009a6518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"
Oct 30 19:58:57.659260 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.659231    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18124ff-856e-437b-875b-897009a6518a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz\" (UID: \"e18124ff-856e-437b-875b-897009a6518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"
Oct 30 19:58:57.659411 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.659330    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18124ff-856e-437b-875b-897009a6518a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz\" (UID: \"e18124ff-856e-437b-875b-897009a6518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"
Oct 30 19:58:57.680100 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.680062    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w75wq\" (UniqueName: \"kubernetes.io/projected/e18124ff-856e-437b-875b-897009a6518a-kube-api-access-w75wq\") pod \"maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz\" (UID: \"e18124ff-856e-437b-875b-897009a6518a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"
Oct 30 19:58:57.743707 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.743633    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:57.751676 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.751653    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"
Oct 30 19:58:57.901345 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.901315    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp"]
Oct 30 19:58:57.905127 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.905095    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp"
Oct 30 19:58:57.920442 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.920422    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-7d43e828-dockercfg-7t54d\""
Oct 30 19:58:57.923373 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.923336    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-a25f47\""
Oct 30 19:58:57.925216 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.925195    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp"]
Oct 30 19:58:57.926413 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.926393    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:57.959709 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.959685    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lknz\" (UniqueName: \"kubernetes.io/projected/4d052c46-3986-468e-aa17-9e0366cf00d9-kube-api-access-2lknz\") pod \"payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp\" (UID: \"4d052c46-3986-468e-aa17-9e0366cf00d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp"
Oct 30 19:58:57.959874 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:57.959853    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d052c46-3986-468e-aa17-9e0366cf00d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp\" (UID: \"4d052c46-3986-468e-aa17-9e0366cf00d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp"
Oct 30 19:58:58.005691 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.005583    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74"]
Oct 30 19:58:58.009539 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.009506    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74"
Oct 30 19:58:58.040682 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.040656    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74"]
Oct 30 19:58:58.054522 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.054500    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"]
Oct 30 19:58:58.055589 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:58:58.055560    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18124ff_856e_437b_875b_897009a6518a.slice/crio-6b0eccb756346819d035e6614e1b078d668ec13cc9603d082554fba1400da011 WatchSource:0}: Error finding container 6b0eccb756346819d035e6614e1b078d668ec13cc9603d082554fba1400da011: Status 404 returned error can't find the container with id 6b0eccb756346819d035e6614e1b078d668ec13cc9603d082554fba1400da011
Oct 30 19:58:58.061049 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.061025    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/043031c4-5bad-4e12-91d8-16693a9ca819-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74\" (UID: \"043031c4-5bad-4e12-91d8-16693a9ca819\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74"
Oct 30 19:58:58.061156 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.061059    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-564rm\" (UniqueName: \"kubernetes.io/projected/043031c4-5bad-4e12-91d8-16693a9ca819-kube-api-access-564rm\") pod \"payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74\" (UID: \"043031c4-5bad-4e12-91d8-16693a9ca819\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74"
Oct 30 19:58:58.061156 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.061094    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d052c46-3986-468e-aa17-9e0366cf00d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp\" (UID: \"4d052c46-3986-468e-aa17-9e0366cf00d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp"
Oct 30 19:58:58.061277 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.061228    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lknz\" (UniqueName: \"kubernetes.io/projected/4d052c46-3986-468e-aa17-9e0366cf00d9-kube-api-access-2lknz\") pod \"payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp\" (UID: \"4d052c46-3986-468e-aa17-9e0366cf00d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp"
Oct 30 19:58:58.061721 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.061696    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d052c46-3986-468e-aa17-9e0366cf00d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp\" (UID: \"4d052c46-3986-468e-aa17-9e0366cf00d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp"
Oct 30 19:58:58.080789 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.080770    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lknz\" (UniqueName: \"kubernetes.io/projected/4d052c46-3986-468e-aa17-9e0366cf00d9-kube-api-access-2lknz\") pod \"payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp\" (UID: \"4d052c46-3986-468e-aa17-9e0366cf00d9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp"
Oct 30 19:58:58.162060 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.162035    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/043031c4-5bad-4e12-91d8-16693a9ca819-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74\" (UID: \"043031c4-5bad-4e12-91d8-16693a9ca819\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74"
Oct 30 19:58:58.162165 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.162067    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-564rm\" (UniqueName: \"kubernetes.io/projected/043031c4-5bad-4e12-91d8-16693a9ca819-kube-api-access-564rm\") pod \"payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74\" (UID: \"043031c4-5bad-4e12-91d8-16693a9ca819\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74"
Oct 30 19:58:58.162766 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.162747    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/043031c4-5bad-4e12-91d8-16693a9ca819-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74\" (UID: \"043031c4-5bad-4e12-91d8-16693a9ca819\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74"
Oct 30 19:58:58.181646 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.181621    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-564rm\" (UniqueName: \"kubernetes.io/projected/043031c4-5bad-4e12-91d8-16693a9ca819-kube-api-access-564rm\") pod \"payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74\" (UID: \"043031c4-5bad-4e12-91d8-16693a9ca819\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74"
Oct 30 19:58:58.218473 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.218451    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp"
Oct 30 19:58:58.323007 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.322973    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74"
Oct 30 19:58:58.508795 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.508769    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp"]
Oct 30 19:58:58.510267 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:58:58.510240    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d052c46_3986_468e_aa17_9e0366cf00d9.slice/crio-b34c0b8a294155bb8533b4d3b85cfd616cf22ada7f2653443400d4fffe7c05d5 WatchSource:0}: Error finding container b34c0b8a294155bb8533b4d3b85cfd616cf22ada7f2653443400d4fffe7c05d5: Status 404 returned error can't find the container with id b34c0b8a294155bb8533b4d3b85cfd616cf22ada7f2653443400d4fffe7c05d5
Oct 30 19:58:58.613295 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.613266    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74"]
Oct 30 19:58:58.613763 ip-10-0-140-248 kubenswrapper[4712]: W1030 19:58:58.613739    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043031c4_5bad_4e12_91d8_16693a9ca819.slice/crio-a801f4d268214e32df963bbd6e54f6ae3e770ca1b945bf601a7db84d588bd36f WatchSource:0}: Error finding container a801f4d268214e32df963bbd6e54f6ae3e770ca1b945bf601a7db84d588bd36f: Status 404 returned error can't find the container with id a801f4d268214e32df963bbd6e54f6ae3e770ca1b945bf601a7db84d588bd36f
Oct 30 19:58:58.668376 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.668339    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz" event={"ID":"e18124ff-856e-437b-875b-897009a6518a","Type":"ContainerStarted","Data":"b79c591bcaa3050fd47d780cb6234a0c9694c7ba1aef08adc03157940ab0c55c"}
Oct 30 19:58:58.668483 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.668405    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz" event={"ID":"e18124ff-856e-437b-875b-897009a6518a","Type":"ContainerStarted","Data":"6b0eccb756346819d035e6614e1b078d668ec13cc9603d082554fba1400da011"}
Oct 30 19:58:58.668539 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.668523    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"
Oct 30 19:58:58.670184 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.670163    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp" event={"ID":"4d052c46-3986-468e-aa17-9e0366cf00d9","Type":"ContainerStarted","Data":"fb855427ff1a3c98283d9bee078659a30df35f37dd9e05c441b78e3298423ddf"}
Oct 30 19:58:58.670295 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.670198    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp" event={"ID":"4d052c46-3986-468e-aa17-9e0366cf00d9","Type":"ContainerStarted","Data":"b34c0b8a294155bb8533b4d3b85cfd616cf22ada7f2653443400d4fffe7c05d5"}
Oct 30 19:58:58.670295 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.670291    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp"
Oct 30 19:58:58.671628 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.671574    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74" event={"ID":"043031c4-5bad-4e12-91d8-16693a9ca819","Type":"ContainerStarted","Data":"a801f4d268214e32df963bbd6e54f6ae3e770ca1b945bf601a7db84d588bd36f"}
Oct 30 19:58:58.717100 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.717053    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz" podStartSLOduration=1.244278712 podStartE2EDuration="1.717041882s" totalImagesPullingTime="472.76317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:58.716077809 +0000 UTC m=+2292.015339938" watchObservedRunningTime="2026-10-30 19:58:58.717041882 +0000 UTC m=+2292.016304011"
Oct 30 19:58:58.764330 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:58.764256    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp" podStartSLOduration=1.764246221 podStartE2EDuration="1.764246221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:58.763736575 +0000 UTC m=+2292.062998704" watchObservedRunningTime="2026-10-30 19:58:58.764246221 +0000 UTC m=+2292.063508350"
Oct 30 19:58:59.278528 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:59.278497    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:58:59.677606 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:59.677564    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74" event={"ID":"043031c4-5bad-4e12-91d8-16693a9ca819","Type":"ContainerStarted","Data":"2fe6676a46f0cb56c5793c2d4bd80ad180d8880a1aa83887741ca740dc7ea169"}
Oct 30 19:58:59.678056 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:59.677919    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74"
Oct 30 19:58:59.717048 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:58:59.717004    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74" podStartSLOduration=2.716994229 podStartE2EDuration="2.716994229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:59.716392536 +0000 UTC m=+2293.015654662" watchObservedRunningTime="2026-10-30 19:58:59.716994229 +0000 UTC m=+2293.016256355"
Oct 30 19:59:00.837666 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:00.837631    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:02.431395 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:02.428038    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:04.019389 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:04.018924    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:04.683758 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:04.683731    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"
Oct 30 19:59:05.601839 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:05.601804    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:06.372328 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:59:06.372292    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:59:07.299513 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:07.299477    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:09.013736 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:09.013696    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:09.665597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:09.665053    4712 patch_prober.go:29] interesting pod/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": read tcp 10.133.0.2:55754->10.133.0.105:15021: read: connection reset by peer" start-of-body=
Oct 30 19:59:09.665597 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:09.665109    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh" podUID="0171ca81-b7fc-41ee-bee3-2cec16b84f48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": read tcp 10.133.0.2:55754->10.133.0.105:15021: read: connection reset by peer"
Oct 30 19:59:09.681211 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:09.681186    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp"
Oct 30 19:59:09.729264 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:09.729232    4712 generic.go:356] "Generic (PLEG): container finished" podID="0171ca81-b7fc-41ee-bee3-2cec16b84f48" containerID="600071442fb4329ec6a13a45590ac7ea3d6a24cdf41950446dc164ff4decbab9" exitCode=0
Oct 30 19:59:09.729431 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:09.729307    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh" event={"ID":"0171ca81-b7fc-41ee-bee3-2cec16b84f48","Type":"ContainerDied","Data":"600071442fb4329ec6a13a45590ac7ea3d6a24cdf41950446dc164ff4decbab9"}
Oct 30 19:59:09.729790 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:09.729754    4712 scope.go:122] "RemoveContainer" containerID="600071442fb4329ec6a13a45590ac7ea3d6a24cdf41950446dc164ff4decbab9"
Oct 30 19:59:09.731070 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:09.731036    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:59:09.731158 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:09.731115    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:59:09.731202 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:09.731157    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:59:10.472723 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:10.472683    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:10.685647 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:10.685619    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74"
Oct 30 19:59:10.740128 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:10.740048    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh" event={"ID":"0171ca81-b7fc-41ee-bee3-2cec16b84f48","Type":"ContainerStarted","Data":"7498ae1b6b4e04189dc9c53b3f655f94336c9de319a2928ef27a4e2f654d8eaa"}
Oct 30 19:59:11.062908 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:11.062835    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:11.359871 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:11.359784    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:59:11.360019 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:11.359995    4712 patch_prober.go:29] interesting pod/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:59:11.360063 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:11.360045    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh" podUID="0171ca81-b7fc-41ee-bee3-2cec16b84f48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:59:11.671590 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:11.671563    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:11.831151 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:11.831082    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:11.989199 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:11.989122    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:12.163429 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:12.163392    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:12.359956 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:12.359866    4712 patch_prober.go:29] interesting pod/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:59:12.359956 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:12.359924    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh" podUID="0171ca81-b7fc-41ee-bee3-2cec16b84f48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:59:12.873635 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:12.873601    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"]
Oct 30 19:59:12.876926 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:12.876902    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:12.917384 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:12.916661    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a25f47\"/\"kube-root-ca.crt\""
Oct 30 19:59:12.917800 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:12.917765    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a25f47\"/\"default-dockercfg-6w52r\""
Oct 30 19:59:12.931280 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:12.929851    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a25f47\"/\"openshift-service-ca.crt\""
Oct 30 19:59:12.931465 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:12.929951    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a25f47\"/\"simulator-single-main-a25f47-kserve-self-signed-certs\""
Oct 30 19:59:12.932593 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:12.932573    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"]
Oct 30 19:59:12.996495 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:12.996454    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:13.047310 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.047283    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-home\") pod \"simulator-single-main-a25f47-kserve-89b789555-wnsn6\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") " pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.047453 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.047334    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-297jv\" (UniqueName: \"kubernetes.io/projected/2f0a21c8-1fd0-470e-893f-0e3799199bf7-kube-api-access-297jv\") pod \"simulator-single-main-a25f47-kserve-89b789555-wnsn6\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") " pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.047453 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.047438    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-model-cache\") pod \"simulator-single-main-a25f47-kserve-89b789555-wnsn6\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") " pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.047564 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.047471    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-tmp-dir\") pod \"simulator-single-main-a25f47-kserve-89b789555-wnsn6\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") " pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.047564 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.047496    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-dshm\") pod \"simulator-single-main-a25f47-kserve-89b789555-wnsn6\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") " pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.047631 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.047581    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f0a21c8-1fd0-470e-893f-0e3799199bf7-tls-certs\") pod \"simulator-single-main-a25f47-kserve-89b789555-wnsn6\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") " pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.148845 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.148754    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-home\") pod \"simulator-single-main-a25f47-kserve-89b789555-wnsn6\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") " pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.148845 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.148804    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-297jv\" (UniqueName: \"kubernetes.io/projected/2f0a21c8-1fd0-470e-893f-0e3799199bf7-kube-api-access-297jv\") pod \"simulator-single-main-a25f47-kserve-89b789555-wnsn6\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") " pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.148845 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.148834    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-model-cache\") pod \"simulator-single-main-a25f47-kserve-89b789555-wnsn6\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") " pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.149149 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.148856    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-tmp-dir\") pod \"simulator-single-main-a25f47-kserve-89b789555-wnsn6\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") " pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.149149 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.148874    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-dshm\") pod \"simulator-single-main-a25f47-kserve-89b789555-wnsn6\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") " pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.149149 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.148922    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f0a21c8-1fd0-470e-893f-0e3799199bf7-tls-certs\") pod \"simulator-single-main-a25f47-kserve-89b789555-wnsn6\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") " pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.149421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.149296    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-home\") pod \"simulator-single-main-a25f47-kserve-89b789555-wnsn6\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") " pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.149421 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.149338    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-model-cache\") pod \"simulator-single-main-a25f47-kserve-89b789555-wnsn6\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") " pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.149875 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.149848    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-tmp-dir\") pod \"simulator-single-main-a25f47-kserve-89b789555-wnsn6\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") " pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.151764 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.151717    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-dshm\") pod \"simulator-single-main-a25f47-kserve-89b789555-wnsn6\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") " pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.152212 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.152191    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f0a21c8-1fd0-470e-893f-0e3799199bf7-tls-certs\") pod \"simulator-single-main-a25f47-kserve-89b789555-wnsn6\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") " pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.169107 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.169082    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-297jv\" (UniqueName: \"kubernetes.io/projected/2f0a21c8-1fd0-470e-893f-0e3799199bf7-kube-api-access-297jv\") pod \"simulator-single-main-a25f47-kserve-89b789555-wnsn6\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") " pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.190958 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.190935    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:13.360443 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.360409    4712 patch_prober.go:29] interesting pod/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:59:13.360673 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.360649    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh" podUID="0171ca81-b7fc-41ee-bee3-2cec16b84f48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:59:13.535400 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.531004    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"]
Oct 30 19:59:13.758455 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.758415    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6" event={"ID":"2f0a21c8-1fd0-470e-893f-0e3799199bf7","Type":"ContainerStarted","Data":"30445303a3f2300b91ef2fb892c9d22fb41e885e6e767d1891b50b29e938bc0a"}
Oct 30 19:59:13.758657 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.758471    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6" event={"ID":"2f0a21c8-1fd0-470e-893f-0e3799199bf7","Type":"ContainerStarted","Data":"f96426eb3fe540d801a606d0e3f71b216e74340a760ab4a895c1fc0254a76777"}
Oct 30 19:59:13.760950 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.760926    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:13.800663 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:13.800582    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6" podStartSLOduration=1.8005703629999998 podStartE2EDuration="1.800570363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:13.798465059 +0000 UTC m=+2307.097727188" watchObservedRunningTime="2026-10-30 19:59:13.800570363 +0000 UTC m=+2307.099832492"
Oct 30 19:59:14.192055 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:14.192024    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:14.359606 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:14.359569    4712 patch_prober.go:29] interesting pod/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:59:14.359771 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:14.359632    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh" podUID="0171ca81-b7fc-41ee-bee3-2cec16b84f48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:59:15.195233 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:15.195195    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:15.360478 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:15.360446    4712 patch_prober.go:29] interesting pod/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:59:15.360653 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:15.360499    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh" podUID="0171ca81-b7fc-41ee-bee3-2cec16b84f48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 19:59:16.362981 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:16.362954    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:59:16.363385 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:16.363229    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:59:16.363976 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:16.363957    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 19:59:16.801959 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:16.801924    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:18.405928 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:18.405893    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:19.995503 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:19.995465    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:21.594163 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:21.594132    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:23.116221 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:23.116187    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:23.191144 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:23.191114    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:23.203762 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:23.203736    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:23.814958 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:23.814930    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 19:59:24.644562 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:24.644530    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:26.202175 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:26.202142    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:27.803580 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:27.803536    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:29.284245 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:29.284210    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:30.782081 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:30.782044    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:32.395791 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:32.395756    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:33.912407 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:33.912375    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:34.735490 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:34.735458    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:35.383616 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:35.383581    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:36.372894 ip-10-0-140-248 kubenswrapper[4712]: E1030 19:59:36.372860    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 19:59:36.966557 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:36.966514    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:38.482902 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:38.482834    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:40.000504 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:40.000474    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:41.495142 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:41.495105    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:43.088637 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:43.088590    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:44.690765 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:44.690726    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:46.270309 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:46.270267    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:47.909176 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:47.909120    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:49.486812 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:49.486770    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:50.980347 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:50.980314    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:52.476489 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:52.476429    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:54.007561 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:54.007521    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:54.493599 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:54.493558    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:55.691237 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:55.691206    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:57.462058 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:57.462006    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:58.809516 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:58.809482    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 19:59:59.412575 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:59.412545    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"]
Oct 30 19:59:59.541950 ip-10-0-140-248 kubenswrapper[4712]: I1030 19:59:59.541917    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:00.378200 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:00.378168    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:02.008718 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:02.008672    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:03.593424 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:03.593357    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:05.075798 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:05.075764    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:05.076204 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:05.076080    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74"]
Oct 30 20:00:05.105265 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:05.105229    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp"]
Oct 30 20:00:05.262643 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:05.262614    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:05.418251 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:05.418218    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:06.331597 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:06.331565    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"]
Oct 30 20:00:06.332398 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:06.332334    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz" podUID="e18124ff-856e-437b-875b-897009a6518a" containerName="maas-api" containerID="cri-o://b79c591bcaa3050fd47d780cb6234a0c9694c7ba1aef08adc03157940ab0c55c" gracePeriod=30
Oct 30 20:00:06.378710 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:00:06.378662    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 20:00:06.452049 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:06.451989    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:07.069640 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.069612    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:07.227576 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.227547    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:07.579324 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.579300    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"
Oct 30 20:00:07.629770 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.629743    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e18124ff-856e-437b-875b-897009a6518a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18124ff-856e-437b-875b-897009a6518a-maas-api-tls\") pod \"e18124ff-856e-437b-875b-897009a6518a\" (UID: \"e18124ff-856e-437b-875b-897009a6518a\") "
Oct 30 20:00:07.629926 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.629849    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e18124ff-856e-437b-875b-897009a6518a-kube-api-access-w75wq\" (UniqueName: \"kubernetes.io/projected/e18124ff-856e-437b-875b-897009a6518a-kube-api-access-w75wq\") pod \"e18124ff-856e-437b-875b-897009a6518a\" (UID: \"e18124ff-856e-437b-875b-897009a6518a\") "
Oct 30 20:00:07.629998 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.629954    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e18124ff-856e-437b-875b-897009a6518a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18124ff-856e-437b-875b-897009a6518a-openshift-service-ca\") pod \"e18124ff-856e-437b-875b-897009a6518a\" (UID: \"e18124ff-856e-437b-875b-897009a6518a\") "
Oct 30 20:00:07.630092 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.630000    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e18124ff-856e-437b-875b-897009a6518a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18124ff-856e-437b-875b-897009a6518a-metrics-certs\") pod \"e18124ff-856e-437b-875b-897009a6518a\" (UID: \"e18124ff-856e-437b-875b-897009a6518a\") "
Oct 30 20:00:07.630431 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.630355    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e18124ff-856e-437b-875b-897009a6518a-openshift-service-ca" pod "e18124ff-856e-437b-875b-897009a6518a" (UID: "e18124ff-856e-437b-875b-897009a6518a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:07.632264 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.632238    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e18124ff-856e-437b-875b-897009a6518a-kube-api-access-w75wq" pod "e18124ff-856e-437b-875b-897009a6518a" (UID: "e18124ff-856e-437b-875b-897009a6518a"). InnerVolumeSpecName "kube-api-access-w75wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:07.632449 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.632422    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e18124ff-856e-437b-875b-897009a6518a-maas-api-tls" pod "e18124ff-856e-437b-875b-897009a6518a" (UID: "e18124ff-856e-437b-875b-897009a6518a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:07.632551 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.632524    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e18124ff-856e-437b-875b-897009a6518a-metrics-certs" pod "e18124ff-856e-437b-875b-897009a6518a" (UID: "e18124ff-856e-437b-875b-897009a6518a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:07.634689 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.634667    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:07.731922 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.731891    4712 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18124ff-856e-437b-875b-897009a6518a-metrics-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:07.731922 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.731920    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18124ff-856e-437b-875b-897009a6518a-maas-api-tls\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:07.732130 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.731930    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w75wq\" (UniqueName: \"kubernetes.io/projected/e18124ff-856e-437b-875b-897009a6518a-kube-api-access-w75wq\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:07.732130 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.731946    4712 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18124ff-856e-437b-875b-897009a6518a-openshift-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:07.999877 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.999787    4712 generic.go:356] "Generic (PLEG): container finished" podID="e18124ff-856e-437b-875b-897009a6518a" containerID="b79c591bcaa3050fd47d780cb6234a0c9694c7ba1aef08adc03157940ab0c55c" exitCode=0
Oct 30 20:00:07.999877 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.999830    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"
Oct 30 20:00:07.999877 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.999849    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz" event={"ID":"e18124ff-856e-437b-875b-897009a6518a","Type":"ContainerDied","Data":"b79c591bcaa3050fd47d780cb6234a0c9694c7ba1aef08adc03157940ab0c55c"}
Oct 30 20:00:08.000133 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.999901    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz" event={"ID":"e18124ff-856e-437b-875b-897009a6518a","Type":"ContainerDied","Data":"6b0eccb756346819d035e6614e1b078d668ec13cc9603d082554fba1400da011"}
Oct 30 20:00:08.000133 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:07.999929    4712 scope.go:122] "RemoveContainer" containerID="b79c591bcaa3050fd47d780cb6234a0c9694c7ba1aef08adc03157940ab0c55c"
Oct 30 20:00:08.008929 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:08.008908    4712 scope.go:122] "RemoveContainer" containerID="b79c591bcaa3050fd47d780cb6234a0c9694c7ba1aef08adc03157940ab0c55c"
Oct 30 20:00:08.009197 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:00:08.009177    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b79c591bcaa3050fd47d780cb6234a0c9694c7ba1aef08adc03157940ab0c55c\": container with ID starting with b79c591bcaa3050fd47d780cb6234a0c9694c7ba1aef08adc03157940ab0c55c not found: ID does not exist" containerID="b79c591bcaa3050fd47d780cb6234a0c9694c7ba1aef08adc03157940ab0c55c"
Oct 30 20:00:08.009249 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:08.009205    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b79c591bcaa3050fd47d780cb6234a0c9694c7ba1aef08adc03157940ab0c55c"} err="failed to get container status \"b79c591bcaa3050fd47d780cb6234a0c9694c7ba1aef08adc03157940ab0c55c\": rpc error: code = NotFound desc = could not find container \"b79c591bcaa3050fd47d780cb6234a0c9694c7ba1aef08adc03157940ab0c55c\": container with ID starting with b79c591bcaa3050fd47d780cb6234a0c9694c7ba1aef08adc03157940ab0c55c not found: ID does not exist"
Oct 30 20:00:08.054075 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:08.054053    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"]
Oct 30 20:00:08.059264 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:08.059245    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a25f47-fdf65999-8qftz"]
Oct 30 20:00:08.851259 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:08.851225    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e18124ff-856e-437b-875b-897009a6518a" path="/var/lib/kubelet/pods/e18124ff-856e-437b-875b-897009a6518a/volumes"
Oct 30 20:00:10.077195 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:10.077160    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74" podUID="043031c4-5bad-4e12-91d8-16693a9ca819" containerName="payload-processing" containerID="cri-o://2fe6676a46f0cb56c5793c2d4bd80ad180d8880a1aa83887741ca740dc7ea169" gracePeriod=25
Oct 30 20:00:10.105533 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:10.105501    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp" podUID="4d052c46-3986-468e-aa17-9e0366cf00d9" containerName="payload-pre-processing" containerID="cri-o://fb855427ff1a3c98283d9bee078659a30df35f37dd9e05c441b78e3298423ddf" gracePeriod=25
Oct 30 20:00:10.682873 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:10.682840    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74" podUID="043031c4-5bad-4e12-91d8-16693a9ca819" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:00:13.680833 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:13.680805    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:13.852061 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:13.852023    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:14.031510 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.031417    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"]
Oct 30 20:00:14.031707 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.031681    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh" podUID="0171ca81-b7fc-41ee-bee3-2cec16b84f48" containerName="istio-proxy" containerID="cri-o://7498ae1b6b4e04189dc9c53b3f655f94336c9de319a2928ef27a4e2f654d8eaa" gracePeriod=30
Oct 30 20:00:14.208821 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.208789    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:14.432517 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.432485    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6" podUID="2f0a21c8-1fd0-470e-893f-0e3799199bf7" containerName="main" containerID="cri-o://30445303a3f2300b91ef2fb892c9d22fb41e885e6e767d1891b50b29e938bc0a" gracePeriod=45
Oct 30 20:00:14.677085 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.677059    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 20:00:14.796698 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.796606    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-tmp-dir\") pod \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") "
Oct 30 20:00:14.796698 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.796690    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f0a21c8-1fd0-470e-893f-0e3799199bf7-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f0a21c8-1fd0-470e-893f-0e3799199bf7-tls-certs\") pod \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") "
Oct 30 20:00:14.797206 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.796726    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0a21c8-1fd0-470e-893f-0e3799199bf7-kube-api-access-297jv\" (UniqueName: \"kubernetes.io/projected/2f0a21c8-1fd0-470e-893f-0e3799199bf7-kube-api-access-297jv\") pod \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") "
Oct 30 20:00:14.797206 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.796827    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-home\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-home\") pod \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") "
Oct 30 20:00:14.797206 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.796873    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-model-cache\") pod \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") "
Oct 30 20:00:14.797206 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.796902    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-tmp-dir" pod "2f0a21c8-1fd0-470e-893f-0e3799199bf7" (UID: "2f0a21c8-1fd0-470e-893f-0e3799199bf7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:14.797206 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.796914    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-dshm\") pod \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\" (UID: \"2f0a21c8-1fd0-470e-893f-0e3799199bf7\") "
Oct 30 20:00:14.797476 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.797333    4712 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-tmp-dir\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:14.797476 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.797398    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-home" pod "2f0a21c8-1fd0-470e-893f-0e3799199bf7" (UID: "2f0a21c8-1fd0-470e-893f-0e3799199bf7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:14.797476 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.797451    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-model-cache" pod "2f0a21c8-1fd0-470e-893f-0e3799199bf7" (UID: "2f0a21c8-1fd0-470e-893f-0e3799199bf7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:14.799395 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.799338    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f0a21c8-1fd0-470e-893f-0e3799199bf7-tls-certs" pod "2f0a21c8-1fd0-470e-893f-0e3799199bf7" (UID: "2f0a21c8-1fd0-470e-893f-0e3799199bf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:14.799551 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.799438    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-dshm" pod "2f0a21c8-1fd0-470e-893f-0e3799199bf7" (UID: "2f0a21c8-1fd0-470e-893f-0e3799199bf7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:14.800612 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.800590    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0a21c8-1fd0-470e-893f-0e3799199bf7-kube-api-access-297jv" pod "2f0a21c8-1fd0-470e-893f-0e3799199bf7" (UID: "2f0a21c8-1fd0-470e-893f-0e3799199bf7"). InnerVolumeSpecName "kube-api-access-297jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:14.898661 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.898633    4712 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f0a21c8-1fd0-470e-893f-0e3799199bf7-tls-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:14.898661 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.898661    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-297jv\" (UniqueName: \"kubernetes.io/projected/2f0a21c8-1fd0-470e-893f-0e3799199bf7-kube-api-access-297jv\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:14.898824 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.898672    4712 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-home\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:14.898824 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.898681    4712 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-model-cache\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:14.898824 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:14.898690    4712 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f0a21c8-1fd0-470e-893f-0e3799199bf7-dshm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:15.039085 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:15.039057    4712 generic.go:356] "Generic (PLEG): container finished" podID="2f0a21c8-1fd0-470e-893f-0e3799199bf7" containerID="30445303a3f2300b91ef2fb892c9d22fb41e885e6e767d1891b50b29e938bc0a" exitCode=0
Oct 30 20:00:15.039231 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:15.039116    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"
Oct 30 20:00:15.039231 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:15.039140    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6" event={"ID":"2f0a21c8-1fd0-470e-893f-0e3799199bf7","Type":"ContainerDied","Data":"30445303a3f2300b91ef2fb892c9d22fb41e885e6e767d1891b50b29e938bc0a"}
Oct 30 20:00:15.039231 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:15.039177    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6" event={"ID":"2f0a21c8-1fd0-470e-893f-0e3799199bf7","Type":"ContainerDied","Data":"f96426eb3fe540d801a606d0e3f71b216e74340a760ab4a895c1fc0254a76777"}
Oct 30 20:00:15.039231 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:15.039195    4712 scope.go:122] "RemoveContainer" containerID="30445303a3f2300b91ef2fb892c9d22fb41e885e6e767d1891b50b29e938bc0a"
Oct 30 20:00:15.047514 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:15.047495    4712 scope.go:122] "RemoveContainer" containerID="30445303a3f2300b91ef2fb892c9d22fb41e885e6e767d1891b50b29e938bc0a"
Oct 30 20:00:15.047791 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:00:15.047770    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30445303a3f2300b91ef2fb892c9d22fb41e885e6e767d1891b50b29e938bc0a\": container with ID starting with 30445303a3f2300b91ef2fb892c9d22fb41e885e6e767d1891b50b29e938bc0a not found: ID does not exist" containerID="30445303a3f2300b91ef2fb892c9d22fb41e885e6e767d1891b50b29e938bc0a"
Oct 30 20:00:15.047835 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:15.047799    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30445303a3f2300b91ef2fb892c9d22fb41e885e6e767d1891b50b29e938bc0a"} err="failed to get container status \"30445303a3f2300b91ef2fb892c9d22fb41e885e6e767d1891b50b29e938bc0a\": rpc error: code = NotFound desc = could not find container \"30445303a3f2300b91ef2fb892c9d22fb41e885e6e767d1891b50b29e938bc0a\": container with ID starting with 30445303a3f2300b91ef2fb892c9d22fb41e885e6e767d1891b50b29e938bc0a not found: ID does not exist"
Oct 30 20:00:15.088648 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:15.088628    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"]
Oct 30 20:00:15.100661 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:15.100638    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-a25f47/simulator-single-main-a25f47-kserve-89b789555-wnsn6"]
Oct 30 20:00:16.365782 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:16.365745    4712 patch_prober.go:29] interesting pod/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:00:16.366216 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:16.365809    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh" podUID="0171ca81-b7fc-41ee-bee3-2cec16b84f48" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:00:16.850954 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:16.850876    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f0a21c8-1fd0-470e-893f-0e3799199bf7" path="/var/lib/kubelet/pods/2f0a21c8-1fd0-470e-893f-0e3799199bf7/volumes"
Oct 30 20:00:18.571213 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.571174    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"]
Oct 30 20:00:18.571772 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.571750    4712 cpu_manager.go:414] "containerMap: removing container" podUID="e18124ff-856e-437b-875b-897009a6518a" containerName="maas-api"
Oct 30 20:00:18.571819 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.571776    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e18124ff-856e-437b-875b-897009a6518a" containerName="maas-api"
Oct 30 20:00:18.571819 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.571802    4712 cpu_manager.go:414] "containerMap: removing container" podUID="2f0a21c8-1fd0-470e-893f-0e3799199bf7" containerName="main"
Oct 30 20:00:18.571819 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.571810    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0a21c8-1fd0-470e-893f-0e3799199bf7" containerName="main"
Oct 30 20:00:18.571950 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.571938    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0a21c8-1fd0-470e-893f-0e3799199bf7" containerName="main"
Oct 30 20:00:18.571984 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.571964    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="e18124ff-856e-437b-875b-897009a6518a" containerName="maas-api"
Oct 30 20:00:18.576920 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.576897    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.593033 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.593006    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"]
Oct 30 20:00:18.602502 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.602478    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d5884a0f-kserve-self-signed-certs\""
Oct 30 20:00:18.631851 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.631822    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c450bf0e-dfda-42df-8d57-717b9e067e21-tls-certs\") pod \"e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") " pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.631950 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.631874    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fqhk\" (UniqueName: \"kubernetes.io/projected/c450bf0e-dfda-42df-8d57-717b9e067e21-kube-api-access-5fqhk\") pod \"e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") " pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.631995 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.631980    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-model-cache\") pod \"e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") " pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.632057 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.632043    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-tmp-dir\") pod \"e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") " pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.632134 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.632113    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-dshm\") pod \"e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") " pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.632181 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.632148    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-home\") pod \"e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") " pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.672784 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.672762    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:18.733513 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.733490    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-model-cache\") pod \"e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") " pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.733617 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.733543    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-tmp-dir\") pod \"e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") " pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.733617 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.733578    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-dshm\") pod \"e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") " pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.733617 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.733597    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-home\") pod \"e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") " pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.733842 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.733627    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c450bf0e-dfda-42df-8d57-717b9e067e21-tls-certs\") pod \"e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") " pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.733842 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.733669    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqhk\" (UniqueName: \"kubernetes.io/projected/c450bf0e-dfda-42df-8d57-717b9e067e21-kube-api-access-5fqhk\") pod \"e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") " pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.734006 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.733981    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-model-cache\") pod \"e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") " pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.734128 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.734098    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-home\") pod \"e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") " pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.734376 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.734348    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-tmp-dir\") pod \"e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") " pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.736104 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.736081    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-dshm\") pod \"e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") " pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.736342 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.736320    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c450bf0e-dfda-42df-8d57-717b9e067e21-tls-certs\") pod \"e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") " pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.754128 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.754101    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fqhk\" (UniqueName: \"kubernetes.io/projected/c450bf0e-dfda-42df-8d57-717b9e067e21-kube-api-access-5fqhk\") pod \"e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") " pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:18.889866 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:18.889838    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:19.184663 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.184634    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"]
Oct 30 20:00:19.185296 ip-10-0-140-248 kubenswrapper[4712]: W1030 20:00:19.185261    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc450bf0e_dfda_42df_8d57_717b9e067e21.slice/crio-5ea28152cc1fbe3f68fbafd224c01c0f3a233599bf36a9e7fc6f093e009ca97d WatchSource:0}: Error finding container 5ea28152cc1fbe3f68fbafd224c01c0f3a233599bf36a9e7fc6f093e009ca97d: Status 404 returned error can't find the container with id 5ea28152cc1fbe3f68fbafd224c01c0f3a233599bf36a9e7fc6f093e009ca97d
Oct 30 20:00:19.274996 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.274968    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 20:00:19.339806 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.339777    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-token\" (UniqueName: \"kubernetes.io/projected/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-token\") pod \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") "
Oct 30 20:00:19.339959 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.339829    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-envoy\") pod \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") "
Oct 30 20:00:19.339959 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.339875    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-data\") pod \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") "
Oct 30 20:00:19.339959 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.339913    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0171ca81-b7fc-41ee-bee3-2cec16b84f48-kube-api-access-n24vw\" (UniqueName: \"kubernetes.io/projected/0171ca81-b7fc-41ee-bee3-2cec16b84f48-kube-api-access-n24vw\") pod \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") "
Oct 30 20:00:19.340135 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.339971    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-credential-socket\") pod \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") "
Oct 30 20:00:19.340135 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.340032    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-workload-certs\") pod \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") "
Oct 30 20:00:19.340135 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.340064    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-workload-socket\") pod \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") "
Oct 30 20:00:19.340135 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.340119    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-podinfo\") pod \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") "
Oct 30 20:00:19.340341 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.340147    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istiod-ca-cert\") pod \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\" (UID: \"0171ca81-b7fc-41ee-bee3-2cec16b84f48\") "
Oct 30 20:00:19.340728 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.340666    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-workload-certs" pod "0171ca81-b7fc-41ee-bee3-2cec16b84f48" (UID: "0171ca81-b7fc-41ee-bee3-2cec16b84f48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:19.340728 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.340684    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-workload-socket" pod "0171ca81-b7fc-41ee-bee3-2cec16b84f48" (UID: "0171ca81-b7fc-41ee-bee3-2cec16b84f48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:19.340728 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.340695    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-credential-socket" pod "0171ca81-b7fc-41ee-bee3-2cec16b84f48" (UID: "0171ca81-b7fc-41ee-bee3-2cec16b84f48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:19.340972 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.340785    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istiod-ca-cert" pod "0171ca81-b7fc-41ee-bee3-2cec16b84f48" (UID: "0171ca81-b7fc-41ee-bee3-2cec16b84f48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:19.340972 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.340849    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-data" pod "0171ca81-b7fc-41ee-bee3-2cec16b84f48" (UID: "0171ca81-b7fc-41ee-bee3-2cec16b84f48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:19.342285 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.342264    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-envoy" pod "0171ca81-b7fc-41ee-bee3-2cec16b84f48" (UID: "0171ca81-b7fc-41ee-bee3-2cec16b84f48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:19.342524 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.342505    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-token" pod "0171ca81-b7fc-41ee-bee3-2cec16b84f48" (UID: "0171ca81-b7fc-41ee-bee3-2cec16b84f48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:19.342616 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.342536    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0171ca81-b7fc-41ee-bee3-2cec16b84f48-kube-api-access-n24vw" pod "0171ca81-b7fc-41ee-bee3-2cec16b84f48" (UID: "0171ca81-b7fc-41ee-bee3-2cec16b84f48"). InnerVolumeSpecName "kube-api-access-n24vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:19.342616 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.342543    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-podinfo" pod "0171ca81-b7fc-41ee-bee3-2cec16b84f48" (UID: "0171ca81-b7fc-41ee-bee3-2cec16b84f48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:00:19.441796 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.441734    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.441796 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.441759    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.441796 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.441773    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.441796 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.441786    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n24vw\" (UniqueName: \"kubernetes.io/projected/0171ca81-b7fc-41ee-bee3-2cec16b84f48-kube-api-access-n24vw\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.441995 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.441798    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.441995 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.441811    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.441995 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.441823    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0171ca81-b7fc-41ee-bee3-2cec16b84f48-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.441995 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.441834    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.441995 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.441847    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0171ca81-b7fc-41ee-bee3-2cec16b84f48-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:19.678871 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:19.678830    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp" podUID="4d052c46-3986-468e-aa17-9e0366cf00d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:00:20.062818 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:20.062739    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm" event={"ID":"c450bf0e-dfda-42df-8d57-717b9e067e21","Type":"ContainerStarted","Data":"0a8424f34b375e62631b7e14100767b50cc8184343d345bd5284e7f211d59826"}
Oct 30 20:00:20.062818 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:20.062781    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm" event={"ID":"c450bf0e-dfda-42df-8d57-717b9e067e21","Type":"ContainerStarted","Data":"5ea28152cc1fbe3f68fbafd224c01c0f3a233599bf36a9e7fc6f093e009ca97d"}
Oct 30 20:00:20.064373 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:20.064343    4712 generic.go:356] "Generic (PLEG): container finished" podID="0171ca81-b7fc-41ee-bee3-2cec16b84f48" containerID="7498ae1b6b4e04189dc9c53b3f655f94336c9de319a2928ef27a4e2f654d8eaa" exitCode=0
Oct 30 20:00:20.064498 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:20.064414    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh" event={"ID":"0171ca81-b7fc-41ee-bee3-2cec16b84f48","Type":"ContainerDied","Data":"7498ae1b6b4e04189dc9c53b3f655f94336c9de319a2928ef27a4e2f654d8eaa"}
Oct 30 20:00:20.064498 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:20.064457    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"
Oct 30 20:00:20.064498 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:20.064469    4712 scope.go:122] "RemoveContainer" containerID="7498ae1b6b4e04189dc9c53b3f655f94336c9de319a2928ef27a4e2f654d8eaa"
Oct 30 20:00:20.064687 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:20.064460    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh" event={"ID":"0171ca81-b7fc-41ee-bee3-2cec16b84f48","Type":"ContainerDied","Data":"9cdeb6df97985a6dbae7c778c1a5fd5e6abb18588d3d8dae8a4dac2dbfc7e7f7"}
Oct 30 20:00:20.073399 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:20.073383    4712 scope.go:122] "RemoveContainer" containerID="600071442fb4329ec6a13a45590ac7ea3d6a24cdf41950446dc164ff4decbab9"
Oct 30 20:00:20.081114 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:20.081096    4712 scope.go:122] "RemoveContainer" containerID="7498ae1b6b4e04189dc9c53b3f655f94336c9de319a2928ef27a4e2f654d8eaa"
Oct 30 20:00:20.081356 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:00:20.081332    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7498ae1b6b4e04189dc9c53b3f655f94336c9de319a2928ef27a4e2f654d8eaa\": container with ID starting with 7498ae1b6b4e04189dc9c53b3f655f94336c9de319a2928ef27a4e2f654d8eaa not found: ID does not exist" containerID="7498ae1b6b4e04189dc9c53b3f655f94336c9de319a2928ef27a4e2f654d8eaa"
Oct 30 20:00:20.081450 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:20.081377    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7498ae1b6b4e04189dc9c53b3f655f94336c9de319a2928ef27a4e2f654d8eaa"} err="failed to get container status \"7498ae1b6b4e04189dc9c53b3f655f94336c9de319a2928ef27a4e2f654d8eaa\": rpc error: code = NotFound desc = could not find container \"7498ae1b6b4e04189dc9c53b3f655f94336c9de319a2928ef27a4e2f654d8eaa\": container with ID starting with 7498ae1b6b4e04189dc9c53b3f655f94336c9de319a2928ef27a4e2f654d8eaa not found: ID does not exist"
Oct 30 20:00:20.081450 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:20.081397    4712 scope.go:122] "RemoveContainer" containerID="600071442fb4329ec6a13a45590ac7ea3d6a24cdf41950446dc164ff4decbab9"
Oct 30 20:00:20.081641 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:00:20.081624    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600071442fb4329ec6a13a45590ac7ea3d6a24cdf41950446dc164ff4decbab9\": container with ID starting with 600071442fb4329ec6a13a45590ac7ea3d6a24cdf41950446dc164ff4decbab9 not found: ID does not exist" containerID="600071442fb4329ec6a13a45590ac7ea3d6a24cdf41950446dc164ff4decbab9"
Oct 30 20:00:20.081692 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:20.081652    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600071442fb4329ec6a13a45590ac7ea3d6a24cdf41950446dc164ff4decbab9"} err="failed to get container status \"600071442fb4329ec6a13a45590ac7ea3d6a24cdf41950446dc164ff4decbab9\": rpc error: code = NotFound desc = could not find container \"600071442fb4329ec6a13a45590ac7ea3d6a24cdf41950446dc164ff4decbab9\": container with ID starting with 600071442fb4329ec6a13a45590ac7ea3d6a24cdf41950446dc164ff4decbab9 not found: ID does not exist"
Oct 30 20:00:20.104582 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:20.104544    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm" podStartSLOduration=2.10453277 podStartE2EDuration="2.10453277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:20.103131413 +0000 UTC m=+2373.402393543" watchObservedRunningTime="2026-10-30 20:00:20.10453277 +0000 UTC m=+2373.403794899"
Oct 30 20:00:20.139666 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:20.139643    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"]
Oct 30 20:00:20.145163 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:20.145142    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a25f47-openshift-default-76f99dd64fdvfbh"]
Oct 30 20:00:20.683590 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:20.683558    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74" podUID="043031c4-5bad-4e12-91d8-16693a9ca819" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:00:20.851153 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:20.851125    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0171ca81-b7fc-41ee-bee3-2cec16b84f48" path="/var/lib/kubelet/pods/0171ca81-b7fc-41ee-bee3-2cec16b84f48/volumes"
Oct 30 20:00:20.890495 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:20.890470    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:28.890705 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:28.890618    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:28.902679 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:28.902657    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:29.127450 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:29.127423    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:00:29.679022 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:29.678978    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp" podUID="4d052c46-3986-468e-aa17-9e0366cf00d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:00:30.349780 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:30.349756    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74"
Oct 30 20:00:30.377915 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:30.377894    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp"
Oct 30 20:00:30.448459 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:30.448436    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d052c46-3986-468e-aa17-9e0366cf00d9-kube-api-access-2lknz\" (UniqueName: \"kubernetes.io/projected/4d052c46-3986-468e-aa17-9e0366cf00d9-kube-api-access-2lknz\") pod \"4d052c46-3986-468e-aa17-9e0366cf00d9\" (UID: \"4d052c46-3986-468e-aa17-9e0366cf00d9\") "
Oct 30 20:00:30.448592 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:30.448543    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d052c46-3986-468e-aa17-9e0366cf00d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d052c46-3986-468e-aa17-9e0366cf00d9-plugins-config-volume\") pod \"4d052c46-3986-468e-aa17-9e0366cf00d9\" (UID: \"4d052c46-3986-468e-aa17-9e0366cf00d9\") "
Oct 30 20:00:30.448592 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:30.448575    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/043031c4-5bad-4e12-91d8-16693a9ca819-kube-api-access-564rm\" (UniqueName: \"kubernetes.io/projected/043031c4-5bad-4e12-91d8-16693a9ca819-kube-api-access-564rm\") pod \"043031c4-5bad-4e12-91d8-16693a9ca819\" (UID: \"043031c4-5bad-4e12-91d8-16693a9ca819\") "
Oct 30 20:00:30.448730 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:30.448612    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/043031c4-5bad-4e12-91d8-16693a9ca819-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/043031c4-5bad-4e12-91d8-16693a9ca819-plugins-config-volume\") pod \"043031c4-5bad-4e12-91d8-16693a9ca819\" (UID: \"043031c4-5bad-4e12-91d8-16693a9ca819\") "
Oct 30 20:00:30.449007 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:30.448899    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d052c46-3986-468e-aa17-9e0366cf00d9-plugins-config-volume" pod "4d052c46-3986-468e-aa17-9e0366cf00d9" (UID: "4d052c46-3986-468e-aa17-9e0366cf00d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:30.449120 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:30.449041    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/043031c4-5bad-4e12-91d8-16693a9ca819-plugins-config-volume" pod "043031c4-5bad-4e12-91d8-16693a9ca819" (UID: "043031c4-5bad-4e12-91d8-16693a9ca819"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:30.450605 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:30.450581    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d052c46-3986-468e-aa17-9e0366cf00d9-kube-api-access-2lknz" pod "4d052c46-3986-468e-aa17-9e0366cf00d9" (UID: "4d052c46-3986-468e-aa17-9e0366cf00d9"). InnerVolumeSpecName "kube-api-access-2lknz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:30.450698 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:30.450650    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043031c4-5bad-4e12-91d8-16693a9ca819-kube-api-access-564rm" pod "043031c4-5bad-4e12-91d8-16693a9ca819" (UID: "043031c4-5bad-4e12-91d8-16693a9ca819"). InnerVolumeSpecName "kube-api-access-564rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:30.550212 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:30.550188    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lknz\" (UniqueName: \"kubernetes.io/projected/4d052c46-3986-468e-aa17-9e0366cf00d9-kube-api-access-2lknz\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:30.550212 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:30.550212    4712 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d052c46-3986-468e-aa17-9e0366cf00d9-plugins-config-volume\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:30.550349 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:30.550221    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-564rm\" (UniqueName: \"kubernetes.io/projected/043031c4-5bad-4e12-91d8-16693a9ca819-kube-api-access-564rm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:30.550349 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:30.550232    4712 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/043031c4-5bad-4e12-91d8-16693a9ca819-plugins-config-volume\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:00:31.126122 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:31.126088    4712 generic.go:356] "Generic (PLEG): container finished" podID="4d052c46-3986-468e-aa17-9e0366cf00d9" containerID="fb855427ff1a3c98283d9bee078659a30df35f37dd9e05c441b78e3298423ddf" exitCode=0
Oct 30 20:00:31.126273 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:31.126140    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp"
Oct 30 20:00:31.126273 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:31.126180    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp" event={"ID":"4d052c46-3986-468e-aa17-9e0366cf00d9","Type":"ContainerDied","Data":"fb855427ff1a3c98283d9bee078659a30df35f37dd9e05c441b78e3298423ddf"}
Oct 30 20:00:31.126273 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:31.126238    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp" event={"ID":"4d052c46-3986-468e-aa17-9e0366cf00d9","Type":"ContainerDied","Data":"b34c0b8a294155bb8533b4d3b85cfd616cf22ada7f2653443400d4fffe7c05d5"}
Oct 30 20:00:31.126273 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:31.126269    4712 scope.go:122] "RemoveContainer" containerID="fb855427ff1a3c98283d9bee078659a30df35f37dd9e05c441b78e3298423ddf"
Oct 30 20:00:31.127639 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:31.127618    4712 generic.go:356] "Generic (PLEG): container finished" podID="043031c4-5bad-4e12-91d8-16693a9ca819" containerID="2fe6676a46f0cb56c5793c2d4bd80ad180d8880a1aa83887741ca740dc7ea169" exitCode=0
Oct 30 20:00:31.127748 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:31.127653    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74" event={"ID":"043031c4-5bad-4e12-91d8-16693a9ca819","Type":"ContainerDied","Data":"2fe6676a46f0cb56c5793c2d4bd80ad180d8880a1aa83887741ca740dc7ea169"}
Oct 30 20:00:31.127748 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:31.127689    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74" event={"ID":"043031c4-5bad-4e12-91d8-16693a9ca819","Type":"ContainerDied","Data":"a801f4d268214e32df963bbd6e54f6ae3e770ca1b945bf601a7db84d588bd36f"}
Oct 30 20:00:31.127874 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:31.127768    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74"
Oct 30 20:00:31.134736 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:31.134719    4712 scope.go:122] "RemoveContainer" containerID="fb855427ff1a3c98283d9bee078659a30df35f37dd9e05c441b78e3298423ddf"
Oct 30 20:00:31.135001 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:00:31.134983    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb855427ff1a3c98283d9bee078659a30df35f37dd9e05c441b78e3298423ddf\": container with ID starting with fb855427ff1a3c98283d9bee078659a30df35f37dd9e05c441b78e3298423ddf not found: ID does not exist" containerID="fb855427ff1a3c98283d9bee078659a30df35f37dd9e05c441b78e3298423ddf"
Oct 30 20:00:31.135041 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:31.135010    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb855427ff1a3c98283d9bee078659a30df35f37dd9e05c441b78e3298423ddf"} err="failed to get container status \"fb855427ff1a3c98283d9bee078659a30df35f37dd9e05c441b78e3298423ddf\": rpc error: code = NotFound desc = could not find container \"fb855427ff1a3c98283d9bee078659a30df35f37dd9e05c441b78e3298423ddf\": container with ID starting with fb855427ff1a3c98283d9bee078659a30df35f37dd9e05c441b78e3298423ddf not found: ID does not exist"
Oct 30 20:00:31.135041 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:31.135025    4712 scope.go:122] "RemoveContainer" containerID="2fe6676a46f0cb56c5793c2d4bd80ad180d8880a1aa83887741ca740dc7ea169"
Oct 30 20:00:31.142410 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:31.142389    4712 scope.go:122] "RemoveContainer" containerID="2fe6676a46f0cb56c5793c2d4bd80ad180d8880a1aa83887741ca740dc7ea169"
Oct 30 20:00:31.142657 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:00:31.142638    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe6676a46f0cb56c5793c2d4bd80ad180d8880a1aa83887741ca740dc7ea169\": container with ID starting with 2fe6676a46f0cb56c5793c2d4bd80ad180d8880a1aa83887741ca740dc7ea169 not found: ID does not exist" containerID="2fe6676a46f0cb56c5793c2d4bd80ad180d8880a1aa83887741ca740dc7ea169"
Oct 30 20:00:31.142722 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:31.142663    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe6676a46f0cb56c5793c2d4bd80ad180d8880a1aa83887741ca740dc7ea169"} err="failed to get container status \"2fe6676a46f0cb56c5793c2d4bd80ad180d8880a1aa83887741ca740dc7ea169\": rpc error: code = NotFound desc = could not find container \"2fe6676a46f0cb56c5793c2d4bd80ad180d8880a1aa83887741ca740dc7ea169\": container with ID starting with 2fe6676a46f0cb56c5793c2d4bd80ad180d8880a1aa83887741ca740dc7ea169 not found: ID does not exist"
Oct 30 20:00:31.178074 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:31.178050    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74"]
Oct 30 20:00:31.184137 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:31.184115    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a25f47-65855c8ccmjq74"]
Oct 30 20:00:31.217401 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:31.217382    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp"]
Oct 30 20:00:31.221191 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:31.221169    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a25f47-57c487drwp"]
Oct 30 20:00:32.850833 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:32.850803    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="043031c4-5bad-4e12-91d8-16693a9ca819" path="/var/lib/kubelet/pods/043031c4-5bad-4e12-91d8-16693a9ca819/volumes"
Oct 30 20:00:32.851213 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:32.851142    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d052c46-3986-468e-aa17-9e0366cf00d9" path="/var/lib/kubelet/pods/4d052c46-3986-468e-aa17-9e0366cf00d9/volumes"
Oct 30 20:00:34.850707 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:34.850666    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:35.939566 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:35.939526    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:36.137253 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.137221    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"]
Oct 30 20:00:36.137679 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.137647    4712 cpu_manager.go:414] "containerMap: removing container" podUID="043031c4-5bad-4e12-91d8-16693a9ca819" containerName="payload-processing"
Oct 30 20:00:36.137679 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.137678    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="043031c4-5bad-4e12-91d8-16693a9ca819" containerName="payload-processing"
Oct 30 20:00:36.138050 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.137692    4712 cpu_manager.go:414] "containerMap: removing container" podUID="0171ca81-b7fc-41ee-bee3-2cec16b84f48" containerName="istio-proxy"
Oct 30 20:00:36.138050 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.137701    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0171ca81-b7fc-41ee-bee3-2cec16b84f48" containerName="istio-proxy"
Oct 30 20:00:36.138050 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.137739    4712 cpu_manager.go:414] "containerMap: removing container" podUID="4d052c46-3986-468e-aa17-9e0366cf00d9" containerName="payload-pre-processing"
Oct 30 20:00:36.138050 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.137900    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d052c46-3986-468e-aa17-9e0366cf00d9" containerName="payload-pre-processing"
Oct 30 20:00:36.138050 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.137995    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="043031c4-5bad-4e12-91d8-16693a9ca819" containerName="payload-processing"
Oct 30 20:00:36.138050 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.138006    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="0171ca81-b7fc-41ee-bee3-2cec16b84f48" containerName="istio-proxy"
Oct 30 20:00:36.138050 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.138017    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d052c46-3986-468e-aa17-9e0366cf00d9" containerName="payload-pre-processing"
Oct 30 20:00:36.142801 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.142782    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.174113 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.174089    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-d5884a0f-kserve-self-signed-certs\""
Oct 30 20:00:36.175797 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.175777    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"]
Oct 30 20:00:36.192489 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.192430    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drs52\" (UniqueName: \"kubernetes.io/projected/5faf89fb-c2aa-4324-8585-ac8a861a455c-kube-api-access-drs52\") pod \"e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") " pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.192577 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.192489    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-home\") pod \"e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") " pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.192577 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.192553    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-model-cache\") pod \"e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") " pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.192648 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.192588    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faf89fb-c2aa-4324-8585-ac8a861a455c-tls-certs\") pod \"e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") " pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.192683 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.192665    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-tmp-dir\") pod \"e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") " pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.192718 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.192703    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-dshm\") pod \"e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") " pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.243939 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.243896    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:36.293344 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.293322    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drs52\" (UniqueName: \"kubernetes.io/projected/5faf89fb-c2aa-4324-8585-ac8a861a455c-kube-api-access-drs52\") pod \"e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") " pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.293462 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.293426    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-home\") pod \"e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") " pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.293462 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.293452    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-model-cache\") pod \"e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") " pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.293542 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.293468    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faf89fb-c2aa-4324-8585-ac8a861a455c-tls-certs\") pod \"e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") " pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.293542 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.293509    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-tmp-dir\") pod \"e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") " pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.293618 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.293547    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-dshm\") pod \"e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") " pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.293896 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.293877    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-model-cache\") pod \"e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") " pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.293963 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.293895    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-tmp-dir\") pod \"e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") " pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.293963 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.293940    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-home\") pod \"e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") " pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.295815 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.295791    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-dshm\") pod \"e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") " pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.296093 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.296076    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faf89fb-c2aa-4324-8585-ac8a861a455c-tls-certs\") pod \"e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") " pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.313331 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.313305    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drs52\" (UniqueName: \"kubernetes.io/projected/5faf89fb-c2aa-4324-8585-ac8a861a455c-kube-api-access-drs52\") pod \"e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") " pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.374548 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:00:36.374497    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 20:00:36.455708 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.455613    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:36.750890 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:36.750867    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"]
Oct 30 20:00:36.752815 ip-10-0-140-248 kubenswrapper[4712]: W1030 20:00:36.752783    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5faf89fb_c2aa_4324_8585_ac8a861a455c.slice/crio-93816e9bf47ddc43269930a12f8ee34ca41afa852536d27e8bb197b23f3d4ced WatchSource:0}: Error finding container 93816e9bf47ddc43269930a12f8ee34ca41afa852536d27e8bb197b23f3d4ced: Status 404 returned error can't find the container with id 93816e9bf47ddc43269930a12f8ee34ca41afa852536d27e8bb197b23f3d4ced
Oct 30 20:00:37.158317 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:37.158283    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd" event={"ID":"5faf89fb-c2aa-4324-8585-ac8a861a455c","Type":"ContainerStarted","Data":"786a7fc05b9b836c845bf87fcfdfc1115eb609d7fb98aad78a2caf9f61bb7d99"}
Oct 30 20:00:37.158736 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:37.158326    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd" event={"ID":"5faf89fb-c2aa-4324-8585-ac8a861a455c","Type":"ContainerStarted","Data":"93816e9bf47ddc43269930a12f8ee34ca41afa852536d27e8bb197b23f3d4ced"}
Oct 30 20:00:37.197478 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:37.197434    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd" podStartSLOduration=1.197421494 podStartE2EDuration="1.197421494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:37.196447917 +0000 UTC m=+2390.495710046" watchObservedRunningTime="2026-10-30 20:00:37.197421494 +0000 UTC m=+2390.496683623"
Oct 30 20:00:37.350164 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:37.350129    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:37.455947 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:37.455873    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:43.354002 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:43.353956    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:00:46.456257 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:46.456226    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:46.468726 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:46.468702    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:00:47.150785 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:47.150749    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 20:00:47.153521 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:47.153500    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 20:00:47.218789 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:00:47.218767    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:01:02.986055 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:02.986021    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"]
Oct 30 20:01:03.124533 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:03.124494    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:01:06.386275 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:01:06.386239    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 20:01:07.241385 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:07.229931    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"]
Oct 30 20:01:08.337434 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:08.337386    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:01:18.006324 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.006289    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm" podUID="c450bf0e-dfda-42df-8d57-717b9e067e21" containerName="main" containerID="cri-o://0a8424f34b375e62631b7e14100767b50cc8184343d345bd5284e7f211d59826" gracePeriod=45
Oct 30 20:01:18.117401 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.117338    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm" podUID="c450bf0e-dfda-42df-8d57-717b9e067e21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.110:8000/ready\": dial tcp 10.133.0.110:8000: connect: connection refused"
Oct 30 20:01:18.248702 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.248677    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:01:18.302426 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.302354    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c450bf0e-dfda-42df-8d57-717b9e067e21-tls-certs\" (UniqueName: \"kubernetes.io/secret/c450bf0e-dfda-42df-8d57-717b9e067e21-tls-certs\") pod \"c450bf0e-dfda-42df-8d57-717b9e067e21\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") "
Oct 30 20:01:18.302426 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.302419    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-home\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-home\") pod \"c450bf0e-dfda-42df-8d57-717b9e067e21\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") "
Oct 30 20:01:18.302567 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.302517    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-model-cache\") pod \"c450bf0e-dfda-42df-8d57-717b9e067e21\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") "
Oct 30 20:01:18.302613 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.302573    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-tmp-dir\") pod \"c450bf0e-dfda-42df-8d57-717b9e067e21\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") "
Oct 30 20:01:18.302656 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.302635    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c450bf0e-dfda-42df-8d57-717b9e067e21-kube-api-access-5fqhk\" (UniqueName: \"kubernetes.io/projected/c450bf0e-dfda-42df-8d57-717b9e067e21-kube-api-access-5fqhk\") pod \"c450bf0e-dfda-42df-8d57-717b9e067e21\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") "
Oct 30 20:01:18.302718 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.302663    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-home" pod "c450bf0e-dfda-42df-8d57-717b9e067e21" (UID: "c450bf0e-dfda-42df-8d57-717b9e067e21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:18.302718 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.302670    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-dshm\") pod \"c450bf0e-dfda-42df-8d57-717b9e067e21\" (UID: \"c450bf0e-dfda-42df-8d57-717b9e067e21\") "
Oct 30 20:01:18.302926 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.302891    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-tmp-dir" pod "c450bf0e-dfda-42df-8d57-717b9e067e21" (UID: "c450bf0e-dfda-42df-8d57-717b9e067e21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:18.303099 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.303055    4712 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-home\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:01:18.303099 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.303076    4712 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-tmp-dir\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:01:18.303230 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.303129    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-model-cache" pod "c450bf0e-dfda-42df-8d57-717b9e067e21" (UID: "c450bf0e-dfda-42df-8d57-717b9e067e21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:18.304581 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.304552    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c450bf0e-dfda-42df-8d57-717b9e067e21-tls-certs" pod "c450bf0e-dfda-42df-8d57-717b9e067e21" (UID: "c450bf0e-dfda-42df-8d57-717b9e067e21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:18.304690 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.304676    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-dshm" pod "c450bf0e-dfda-42df-8d57-717b9e067e21" (UID: "c450bf0e-dfda-42df-8d57-717b9e067e21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:18.304800 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.304786    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c450bf0e-dfda-42df-8d57-717b9e067e21-kube-api-access-5fqhk" pod "c450bf0e-dfda-42df-8d57-717b9e067e21" (UID: "c450bf0e-dfda-42df-8d57-717b9e067e21"). InnerVolumeSpecName "kube-api-access-5fqhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:18.399340 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.399316    4712 generic.go:356] "Generic (PLEG): container finished" podID="c450bf0e-dfda-42df-8d57-717b9e067e21" containerID="0a8424f34b375e62631b7e14100767b50cc8184343d345bd5284e7f211d59826" exitCode=0
Oct 30 20:01:18.399464 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.399372    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"
Oct 30 20:01:18.399464 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.399398    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm" event={"ID":"c450bf0e-dfda-42df-8d57-717b9e067e21","Type":"ContainerDied","Data":"0a8424f34b375e62631b7e14100767b50cc8184343d345bd5284e7f211d59826"}
Oct 30 20:01:18.399464 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.399438    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm" event={"ID":"c450bf0e-dfda-42df-8d57-717b9e067e21","Type":"ContainerDied","Data":"5ea28152cc1fbe3f68fbafd224c01c0f3a233599bf36a9e7fc6f093e009ca97d"}
Oct 30 20:01:18.399464 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.399454    4712 scope.go:122] "RemoveContainer" containerID="0a8424f34b375e62631b7e14100767b50cc8184343d345bd5284e7f211d59826"
Oct 30 20:01:18.403647 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.403628    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fqhk\" (UniqueName: \"kubernetes.io/projected/c450bf0e-dfda-42df-8d57-717b9e067e21-kube-api-access-5fqhk\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:01:18.403647 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.403647    4712 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-dshm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:01:18.403789 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.403658    4712 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c450bf0e-dfda-42df-8d57-717b9e067e21-tls-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:01:18.403789 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.403666    4712 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c450bf0e-dfda-42df-8d57-717b9e067e21-model-cache\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:01:18.407632 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.407614    4712 scope.go:122] "RemoveContainer" containerID="0a8424f34b375e62631b7e14100767b50cc8184343d345bd5284e7f211d59826"
Oct 30 20:01:18.407881 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:01:18.407863    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8424f34b375e62631b7e14100767b50cc8184343d345bd5284e7f211d59826\": container with ID starting with 0a8424f34b375e62631b7e14100767b50cc8184343d345bd5284e7f211d59826 not found: ID does not exist" containerID="0a8424f34b375e62631b7e14100767b50cc8184343d345bd5284e7f211d59826"
Oct 30 20:01:18.407932 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.407888    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8424f34b375e62631b7e14100767b50cc8184343d345bd5284e7f211d59826"} err="failed to get container status \"0a8424f34b375e62631b7e14100767b50cc8184343d345bd5284e7f211d59826\": rpc error: code = NotFound desc = could not find container \"0a8424f34b375e62631b7e14100767b50cc8184343d345bd5284e7f211d59826\": container with ID starting with 0a8424f34b375e62631b7e14100767b50cc8184343d345bd5284e7f211d59826 not found: ID does not exist"
Oct 30 20:01:18.448796 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.448772    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"]
Oct 30 20:01:18.452962 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.452940    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d5884a0f-kserve-7c7c74b6cc-vtkcm"]
Oct 30 20:01:18.850866 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:18.850835    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c450bf0e-dfda-42df-8d57-717b9e067e21" path="/var/lib/kubelet/pods/c450bf0e-dfda-42df-8d57-717b9e067e21/volumes"
Oct 30 20:01:22.304259 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.304217    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd" podUID="5faf89fb-c2aa-4324-8585-ac8a861a455c" containerName="main" containerID="cri-o://786a7fc05b9b836c845bf87fcfdfc1115eb609d7fb98aad78a2caf9f61bb7d99" gracePeriod=45
Oct 30 20:01:22.423154 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.423124    4712 generic.go:356] "Generic (PLEG): container finished" podID="5faf89fb-c2aa-4324-8585-ac8a861a455c" containerID="786a7fc05b9b836c845bf87fcfdfc1115eb609d7fb98aad78a2caf9f61bb7d99" exitCode=0
Oct 30 20:01:22.423513 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.423483    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd" event={"ID":"5faf89fb-c2aa-4324-8585-ac8a861a455c","Type":"ContainerDied","Data":"786a7fc05b9b836c845bf87fcfdfc1115eb609d7fb98aad78a2caf9f61bb7d99"}
Oct 30 20:01:22.547907 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.547878    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:01:22.647647 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.646676    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5faf89fb-c2aa-4324-8585-ac8a861a455c-tls-certs\" (UniqueName: \"kubernetes.io/secret/5faf89fb-c2aa-4324-8585-ac8a861a455c-tls-certs\") pod \"5faf89fb-c2aa-4324-8585-ac8a861a455c\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") "
Oct 30 20:01:22.647647 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.646791    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-tmp-dir\") pod \"5faf89fb-c2aa-4324-8585-ac8a861a455c\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") "
Oct 30 20:01:22.647647 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.646830    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5faf89fb-c2aa-4324-8585-ac8a861a455c-kube-api-access-drs52\" (UniqueName: \"kubernetes.io/projected/5faf89fb-c2aa-4324-8585-ac8a861a455c-kube-api-access-drs52\") pod \"5faf89fb-c2aa-4324-8585-ac8a861a455c\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") "
Oct 30 20:01:22.647647 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.646864    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-home\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-home\") pod \"5faf89fb-c2aa-4324-8585-ac8a861a455c\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") "
Oct 30 20:01:22.647647 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.646937    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-model-cache\") pod \"5faf89fb-c2aa-4324-8585-ac8a861a455c\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") "
Oct 30 20:01:22.647647 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.646992    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-dshm\") pod \"5faf89fb-c2aa-4324-8585-ac8a861a455c\" (UID: \"5faf89fb-c2aa-4324-8585-ac8a861a455c\") "
Oct 30 20:01:22.672336 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.671663    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5faf89fb-c2aa-4324-8585-ac8a861a455c-kube-api-access-drs52" pod "5faf89fb-c2aa-4324-8585-ac8a861a455c" (UID: "5faf89fb-c2aa-4324-8585-ac8a861a455c"). InnerVolumeSpecName "kube-api-access-drs52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:22.672336 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.671877    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-home" pod "5faf89fb-c2aa-4324-8585-ac8a861a455c" (UID: "5faf89fb-c2aa-4324-8585-ac8a861a455c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:22.672336 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.672070    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-tmp-dir" pod "5faf89fb-c2aa-4324-8585-ac8a861a455c" (UID: "5faf89fb-c2aa-4324-8585-ac8a861a455c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:22.672336 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.672123    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-model-cache" pod "5faf89fb-c2aa-4324-8585-ac8a861a455c" (UID: "5faf89fb-c2aa-4324-8585-ac8a861a455c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:22.682792 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.682747    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-dshm" pod "5faf89fb-c2aa-4324-8585-ac8a861a455c" (UID: "5faf89fb-c2aa-4324-8585-ac8a861a455c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:22.684725 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.684683    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5faf89fb-c2aa-4324-8585-ac8a861a455c-tls-certs" pod "5faf89fb-c2aa-4324-8585-ac8a861a455c" (UID: "5faf89fb-c2aa-4324-8585-ac8a861a455c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:22.751944 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.751840    4712 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-home\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:01:22.751944 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.751871    4712 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-model-cache\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:01:22.751944 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.751885    4712 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-dshm\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:01:22.751944 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.751899    4712 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faf89fb-c2aa-4324-8585-ac8a861a455c-tls-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:01:22.751944 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.751911    4712 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faf89fb-c2aa-4324-8585-ac8a861a455c-tmp-dir\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:01:22.751944 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:22.751923    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drs52\" (UniqueName: \"kubernetes.io/projected/5faf89fb-c2aa-4324-8585-ac8a861a455c-kube-api-access-drs52\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:01:23.452386 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:23.444618    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd" event={"ID":"5faf89fb-c2aa-4324-8585-ac8a861a455c","Type":"ContainerDied","Data":"93816e9bf47ddc43269930a12f8ee34ca41afa852536d27e8bb197b23f3d4ced"}
Oct 30 20:01:23.452386 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:23.444680    4712 scope.go:122] "RemoveContainer" containerID="786a7fc05b9b836c845bf87fcfdfc1115eb609d7fb98aad78a2caf9f61bb7d99"
Oct 30 20:01:23.452386 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:23.444812    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"
Oct 30 20:01:23.523794 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:23.523762    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"]
Oct 30 20:01:23.525824 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:23.525798    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-d5884a0f-kserve-9877dc68f-jq5vd"]
Oct 30 20:01:24.584855 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:24.584822    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:01:24.850681 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:24.850609    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5faf89fb-c2aa-4324-8585-ac8a861a455c" path="/var/lib/kubelet/pods/5faf89fb-c2aa-4324-8585-ac8a861a455c/volumes"
Oct 30 20:01:34.959194 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:34.959158    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:01:36.373572 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:01:36.373533    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 20:01:46.944918 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:46.944890    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"]
Oct 30 20:01:46.945272 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:46.945211    4712 cpu_manager.go:414] "containerMap: removing container" podUID="5faf89fb-c2aa-4324-8585-ac8a861a455c" containerName="main"
Oct 30 20:01:46.945272 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:46.945221    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5faf89fb-c2aa-4324-8585-ac8a861a455c" containerName="main"
Oct 30 20:01:46.945272 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:46.945236    4712 cpu_manager.go:414] "containerMap: removing container" podUID="c450bf0e-dfda-42df-8d57-717b9e067e21" containerName="main"
Oct 30 20:01:46.945272 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:46.945241    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c450bf0e-dfda-42df-8d57-717b9e067e21" containerName="main"
Oct 30 20:01:46.945272 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:46.945253    4712 cpu_manager.go:414] "containerMap: removing container" podUID="0171ca81-b7fc-41ee-bee3-2cec16b84f48" containerName="istio-proxy"
Oct 30 20:01:46.945272 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:46.945258    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0171ca81-b7fc-41ee-bee3-2cec16b84f48" containerName="istio-proxy"
Oct 30 20:01:46.945520 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:46.945321    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="5faf89fb-c2aa-4324-8585-ac8a861a455c" containerName="main"
Oct 30 20:01:46.945520 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:46.945330    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="c450bf0e-dfda-42df-8d57-717b9e067e21" containerName="main"
Oct 30 20:01:46.945520 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:46.945336    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="0171ca81-b7fc-41ee-bee3-2cec16b84f48" containerName="istio-proxy"
Oct 30 20:01:46.948261 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:46.948243    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"
Oct 30 20:01:46.978028 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:46.978003    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"]
Oct 30 20:01:47.108661 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:47.108632    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b4d803-9fa8-438c-9c08-094eb6405987-openshift-service-ca\") pod \"maas-api-9dcd87b65-r5rpr\" (UID: \"84b4d803-9fa8-438c-9c08-094eb6405987\") " pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"
Oct 30 20:01:47.108816 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:47.108703    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwd9r\" (UniqueName: \"kubernetes.io/projected/84b4d803-9fa8-438c-9c08-094eb6405987-kube-api-access-gwd9r\") pod \"maas-api-9dcd87b65-r5rpr\" (UID: \"84b4d803-9fa8-438c-9c08-094eb6405987\") " pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"
Oct 30 20:01:47.108816 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:47.108754    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b4d803-9fa8-438c-9c08-094eb6405987-maas-api-tls\") pod \"maas-api-9dcd87b65-r5rpr\" (UID: \"84b4d803-9fa8-438c-9c08-094eb6405987\") " pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"
Oct 30 20:01:47.108816 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:47.108778    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84b4d803-9fa8-438c-9c08-094eb6405987-metrics-certs\") pod \"maas-api-9dcd87b65-r5rpr\" (UID: \"84b4d803-9fa8-438c-9c08-094eb6405987\") " pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"
Oct 30 20:01:47.210032 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:47.209956    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b4d803-9fa8-438c-9c08-094eb6405987-openshift-service-ca\") pod \"maas-api-9dcd87b65-r5rpr\" (UID: \"84b4d803-9fa8-438c-9c08-094eb6405987\") " pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"
Oct 30 20:01:47.210032 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:47.210026    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwd9r\" (UniqueName: \"kubernetes.io/projected/84b4d803-9fa8-438c-9c08-094eb6405987-kube-api-access-gwd9r\") pod \"maas-api-9dcd87b65-r5rpr\" (UID: \"84b4d803-9fa8-438c-9c08-094eb6405987\") " pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"
Oct 30 20:01:47.210235 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:47.210053    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b4d803-9fa8-438c-9c08-094eb6405987-maas-api-tls\") pod \"maas-api-9dcd87b65-r5rpr\" (UID: \"84b4d803-9fa8-438c-9c08-094eb6405987\") " pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"
Oct 30 20:01:47.210235 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:47.210075    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84b4d803-9fa8-438c-9c08-094eb6405987-metrics-certs\") pod \"maas-api-9dcd87b65-r5rpr\" (UID: \"84b4d803-9fa8-438c-9c08-094eb6405987\") " pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"
Oct 30 20:01:47.210611 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:47.210590    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b4d803-9fa8-438c-9c08-094eb6405987-openshift-service-ca\") pod \"maas-api-9dcd87b65-r5rpr\" (UID: \"84b4d803-9fa8-438c-9c08-094eb6405987\") " pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"
Oct 30 20:01:47.212392 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:47.212355    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b4d803-9fa8-438c-9c08-094eb6405987-maas-api-tls\") pod \"maas-api-9dcd87b65-r5rpr\" (UID: \"84b4d803-9fa8-438c-9c08-094eb6405987\") " pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"
Oct 30 20:01:47.212494 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:47.212476    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84b4d803-9fa8-438c-9c08-094eb6405987-metrics-certs\") pod \"maas-api-9dcd87b65-r5rpr\" (UID: \"84b4d803-9fa8-438c-9c08-094eb6405987\") " pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"
Oct 30 20:01:47.229959 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:47.229938    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwd9r\" (UniqueName: \"kubernetes.io/projected/84b4d803-9fa8-438c-9c08-094eb6405987-kube-api-access-gwd9r\") pod \"maas-api-9dcd87b65-r5rpr\" (UID: \"84b4d803-9fa8-438c-9c08-094eb6405987\") " pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"
Oct 30 20:01:47.260922 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:47.260898    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"
Oct 30 20:01:47.556456 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:47.556419    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"]
Oct 30 20:01:47.557113 ip-10-0-140-248 kubenswrapper[4712]: W1030 20:01:47.557079    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b4d803_9fa8_438c_9c08_094eb6405987.slice/crio-9aec8b8e132516d2d37c00a1ead882f60d4b4ec194573a6702679b891c8e7d85 WatchSource:0}: Error finding container 9aec8b8e132516d2d37c00a1ead882f60d4b4ec194573a6702679b891c8e7d85: Status 404 returned error can't find the container with id 9aec8b8e132516d2d37c00a1ead882f60d4b4ec194573a6702679b891c8e7d85
Oct 30 20:01:47.558650 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:47.558630    4712 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:01:48.561107 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:48.561022    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr" event={"ID":"84b4d803-9fa8-438c-9c08-094eb6405987","Type":"ContainerStarted","Data":"e4e394a7788a80002561c9376948e3ed3e80b68d1868b723ad5d2bcd61343573"}
Oct 30 20:01:48.561107 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:48.561071    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr" event={"ID":"84b4d803-9fa8-438c-9c08-094eb6405987","Type":"ContainerStarted","Data":"9aec8b8e132516d2d37c00a1ead882f60d4b4ec194573a6702679b891c8e7d85"}
Oct 30 20:01:48.561575 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:48.561211    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"
Oct 30 20:01:48.601879 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:48.601821    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr" podStartSLOduration=2.262765441 podStartE2EDuration="2.601808863s" totalImagesPullingTime="339.043422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:48.600383042 +0000 UTC m=+2461.899645162" watchObservedRunningTime="2026-10-30 20:01:48.601808863 +0000 UTC m=+2461.901070992"
Oct 30 20:01:50.295658 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:50.295624    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:01:53.660729 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:53.660693    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:01:53.800268 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:53.800235    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:01:54.189434 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.189403    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"]
Oct 30 20:01:54.204501 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.204463    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.217026 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.217001    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"]
Oct 30 20:01:54.226183 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.226160    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c8feb399-openshift-default-dockercfg-k2jdm\""
Oct 30 20:01:54.269435 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.269410    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b26d2c-afba-44d7-8323-555420014ee1-istiod-ca-cert\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.269548 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.269451    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b26d2c-afba-44d7-8323-555420014ee1-istio-token\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.269548 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.269471    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-workload-certs\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.269637 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.269613    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-credential-socket\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.269682 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.269668    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jxfw\" (UniqueName: \"kubernetes.io/projected/a9b26d2c-afba-44d7-8323-555420014ee1-kube-api-access-7jxfw\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.269768 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.269751    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-istio-envoy\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.269809 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.269788    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-istio-data\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.269846 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.269828    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-workload-socket\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.269883 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.269864    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b26d2c-afba-44d7-8323-555420014ee1-istio-podinfo\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.289130 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.289097    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:01:54.371220 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.371184    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b26d2c-afba-44d7-8323-555420014ee1-istiod-ca-cert\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.371431 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.371229    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b26d2c-afba-44d7-8323-555420014ee1-istio-token\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.371431 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.371250    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-workload-certs\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.371431 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.371276    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-credential-socket\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.371431 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.371302    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jxfw\" (UniqueName: \"kubernetes.io/projected/a9b26d2c-afba-44d7-8323-555420014ee1-kube-api-access-7jxfw\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.371431 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.371322    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-istio-envoy\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.371431 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.371358    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-istio-data\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.371768 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.371433    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-workload-socket\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.371768 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.371458    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b26d2c-afba-44d7-8323-555420014ee1-istio-podinfo\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.371768 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.371750    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-workload-certs\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.371920 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.371789    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-istio-data\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.371920 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.371862    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-workload-socket\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.372033 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.371998    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-credential-socket\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.372033 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.372006    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b26d2c-afba-44d7-8323-555420014ee1-istiod-ca-cert\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.373808 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.373776    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-istio-envoy\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.374105 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.374084    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b26d2c-afba-44d7-8323-555420014ee1-istio-podinfo\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.392473 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.392451    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b26d2c-afba-44d7-8323-555420014ee1-istio-token\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.392609 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.392497    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jxfw\" (UniqueName: \"kubernetes.io/projected/a9b26d2c-afba-44d7-8323-555420014ee1-kube-api-access-7jxfw\") pod \"e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") " pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.521563 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.521473    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:54.526755 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.526722    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:01:54.573942 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.573728    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-9dcd87b65-r5rpr"
Oct 30 20:01:54.645426 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.645389    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"]
Oct 30 20:01:54.645607 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.645589    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4" podUID="a92f3725-c731-4b4c-a272-61442b9ee11c" containerName="maas-api" containerID="cri-o://7c542a958105b40c2f694b20e47ce218cc38b0809bbcd8cd125c21c62651d601" gracePeriod=30
Oct 30 20:01:54.830039 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.830011    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"]
Oct 30 20:01:54.830805 ip-10-0-140-248 kubenswrapper[4712]: W1030 20:01:54.830773    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b26d2c_afba_44d7_8323_555420014ee1.slice/crio-7a20d67f7c1541dbd8e417b7a1725e922528c479b8d376b61390d77464be30fd WatchSource:0}: Error finding container 7a20d67f7c1541dbd8e417b7a1725e922528c479b8d376b61390d77464be30fd: Status 404 returned error can't find the container with id 7a20d67f7c1541dbd8e417b7a1725e922528c479b8d376b61390d77464be30fd
Oct 30 20:01:54.833784 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.833751    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:01:54.833874 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.833816    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:01:54.833874 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:54.833849    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:01:55.596379 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:55.596334    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2" event={"ID":"a9b26d2c-afba-44d7-8323-555420014ee1","Type":"ContainerStarted","Data":"b4f790a78b27ff3b1a747d272ebdae8334c937a203a5f60cd2089e94cdd1125d"}
Oct 30 20:01:55.596560 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:55.596410    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2" event={"ID":"a9b26d2c-afba-44d7-8323-555420014ee1","Type":"ContainerStarted","Data":"7a20d67f7c1541dbd8e417b7a1725e922528c479b8d376b61390d77464be30fd"}
Oct 30 20:01:55.640209 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:55.640165    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2" podStartSLOduration=1.640153063 podStartE2EDuration="1.640153063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:55.639491695 +0000 UTC m=+2468.938753834" watchObservedRunningTime="2026-10-30 20:01:55.640153063 +0000 UTC m=+2468.939415191"
Oct 30 20:01:55.900173 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:55.900143    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"
Oct 30 20:01:55.986254 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:55.986220    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a92f3725-c731-4b4c-a272-61442b9ee11c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a92f3725-c731-4b4c-a272-61442b9ee11c-maas-api-tls\") pod \"a92f3725-c731-4b4c-a272-61442b9ee11c\" (UID: \"a92f3725-c731-4b4c-a272-61442b9ee11c\") "
Oct 30 20:01:55.986407 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:55.986298    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a92f3725-c731-4b4c-a272-61442b9ee11c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92f3725-c731-4b4c-a272-61442b9ee11c-metrics-certs\") pod \"a92f3725-c731-4b4c-a272-61442b9ee11c\" (UID: \"a92f3725-c731-4b4c-a272-61442b9ee11c\") "
Oct 30 20:01:55.986407 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:55.986400    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a92f3725-c731-4b4c-a272-61442b9ee11c-kube-api-access-rcsrk\" (UniqueName: \"kubernetes.io/projected/a92f3725-c731-4b4c-a272-61442b9ee11c-kube-api-access-rcsrk\") pod \"a92f3725-c731-4b4c-a272-61442b9ee11c\" (UID: \"a92f3725-c731-4b4c-a272-61442b9ee11c\") "
Oct 30 20:01:55.986526 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:55.986425    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a92f3725-c731-4b4c-a272-61442b9ee11c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92f3725-c731-4b4c-a272-61442b9ee11c-openshift-service-ca\") pod \"a92f3725-c731-4b4c-a272-61442b9ee11c\" (UID: \"a92f3725-c731-4b4c-a272-61442b9ee11c\") "
Oct 30 20:01:55.986826 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:55.986797    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a92f3725-c731-4b4c-a272-61442b9ee11c-openshift-service-ca" pod "a92f3725-c731-4b4c-a272-61442b9ee11c" (UID: "a92f3725-c731-4b4c-a272-61442b9ee11c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:55.988327 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:55.988300    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a92f3725-c731-4b4c-a272-61442b9ee11c-maas-api-tls" pod "a92f3725-c731-4b4c-a272-61442b9ee11c" (UID: "a92f3725-c731-4b4c-a272-61442b9ee11c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:55.988457 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:55.988357    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a92f3725-c731-4b4c-a272-61442b9ee11c-metrics-certs" pod "a92f3725-c731-4b4c-a272-61442b9ee11c" (UID: "a92f3725-c731-4b4c-a272-61442b9ee11c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:55.988457 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:55.988379    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92f3725-c731-4b4c-a272-61442b9ee11c-kube-api-access-rcsrk" pod "a92f3725-c731-4b4c-a272-61442b9ee11c" (UID: "a92f3725-c731-4b4c-a272-61442b9ee11c"). InnerVolumeSpecName "kube-api-access-rcsrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:56.088014 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:56.087987    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcsrk\" (UniqueName: \"kubernetes.io/projected/a92f3725-c731-4b4c-a272-61442b9ee11c-kube-api-access-rcsrk\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:01:56.088014 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:56.088013    4712 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a92f3725-c731-4b4c-a272-61442b9ee11c-openshift-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:01:56.088181 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:56.088025    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a92f3725-c731-4b4c-a272-61442b9ee11c-maas-api-tls\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:01:56.088181 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:56.088034    4712 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a92f3725-c731-4b4c-a272-61442b9ee11c-metrics-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:01:56.521614 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:56.521580    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:56.526564 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:56.526537    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:56.602270 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:56.602238    4712 generic.go:356] "Generic (PLEG): container finished" podID="a92f3725-c731-4b4c-a272-61442b9ee11c" containerID="7c542a958105b40c2f694b20e47ce218cc38b0809bbcd8cd125c21c62651d601" exitCode=0
Oct 30 20:01:56.602412 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:56.602282    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"
Oct 30 20:01:56.602412 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:56.602322    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4" event={"ID":"a92f3725-c731-4b4c-a272-61442b9ee11c","Type":"ContainerDied","Data":"7c542a958105b40c2f694b20e47ce218cc38b0809bbcd8cd125c21c62651d601"}
Oct 30 20:01:56.602412 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:56.602386    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4" event={"ID":"a92f3725-c731-4b4c-a272-61442b9ee11c","Type":"ContainerDied","Data":"2bcde6ddf9101a5f59ccfb3c12eb99e9eff4bf37fcd9f4bbf7daacd1918c2104"}
Oct 30 20:01:56.602412 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:56.602410    4712 scope.go:122] "RemoveContainer" containerID="7c542a958105b40c2f694b20e47ce218cc38b0809bbcd8cd125c21c62651d601"
Oct 30 20:01:56.602658 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:56.602634    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:56.603944 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:56.603917    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:01:56.612595 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:56.612581    4712 scope.go:122] "RemoveContainer" containerID="7c542a958105b40c2f694b20e47ce218cc38b0809bbcd8cd125c21c62651d601"
Oct 30 20:01:56.612871 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:01:56.612846    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c542a958105b40c2f694b20e47ce218cc38b0809bbcd8cd125c21c62651d601\": container with ID starting with 7c542a958105b40c2f694b20e47ce218cc38b0809bbcd8cd125c21c62651d601 not found: ID does not exist" containerID="7c542a958105b40c2f694b20e47ce218cc38b0809bbcd8cd125c21c62651d601"
Oct 30 20:01:56.612953 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:56.612880    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c542a958105b40c2f694b20e47ce218cc38b0809bbcd8cd125c21c62651d601"} err="failed to get container status \"7c542a958105b40c2f694b20e47ce218cc38b0809bbcd8cd125c21c62651d601\": rpc error: code = NotFound desc = could not find container \"7c542a958105b40c2f694b20e47ce218cc38b0809bbcd8cd125c21c62651d601\": container with ID starting with 7c542a958105b40c2f694b20e47ce218cc38b0809bbcd8cd125c21c62651d601 not found: ID does not exist"
Oct 30 20:01:56.694198 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:56.694171    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"]
Oct 30 20:01:56.698397 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:56.698349    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f5c657c85-pl4k4"]
Oct 30 20:01:56.851133 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:56.851061    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a92f3725-c731-4b4c-a272-61442b9ee11c" path="/var/lib/kubelet/pods/a92f3725-c731-4b4c-a272-61442b9ee11c/volumes"
Oct 30 20:01:57.343609 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:57.343575    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:01:57.501868 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:01:57.501840    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:00.073948 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.073877    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:00.119605 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.119569    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj"]
Oct 30 20:02:00.119933 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.119921    4712 cpu_manager.go:414] "containerMap: removing container" podUID="a92f3725-c731-4b4c-a272-61442b9ee11c" containerName="maas-api"
Oct 30 20:02:00.119972 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.119935    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92f3725-c731-4b4c-a272-61442b9ee11c" containerName="maas-api"
Oct 30 20:02:00.120020 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.120009    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="a92f3725-c731-4b4c-a272-61442b9ee11c" containerName="maas-api"
Oct 30 20:02:00.124541 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.124510    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj"
Oct 30 20:02:00.127795 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.127769    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9"]
Oct 30 20:02:00.130928 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.130909    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9"
Oct 30 20:02:00.156918 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.156894    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c8feb399-dockercfg-x4dpn\""
Oct 30 20:02:00.157600 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.157578    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj"]
Oct 30 20:02:00.159628 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.159607    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9"]
Oct 30 20:02:00.163396 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.163377    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c8feb399\""
Oct 30 20:02:00.225436 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.225412    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lk8r\" (UniqueName: \"kubernetes.io/projected/9e627b8b-63bc-4af4-9743-9301f554a904-kube-api-access-5lk8r\") pod \"payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj\" (UID: \"9e627b8b-63bc-4af4-9743-9301f554a904\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj"
Oct 30 20:02:00.225560 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.225442    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e627b8b-63bc-4af4-9743-9301f554a904-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj\" (UID: \"9e627b8b-63bc-4af4-9743-9301f554a904\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj"
Oct 30 20:02:00.225560 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.225464    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44lsv\" (UniqueName: \"kubernetes.io/projected/c089c321-fd6b-4760-9f13-15ae59bd67ab-kube-api-access-44lsv\") pod \"payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9\" (UID: \"c089c321-fd6b-4760-9f13-15ae59bd67ab\") " pod="openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9"
Oct 30 20:02:00.225560 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.225484    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c089c321-fd6b-4760-9f13-15ae59bd67ab-plugins-config-volume\") pod \"payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9\" (UID: \"c089c321-fd6b-4760-9f13-15ae59bd67ab\") " pod="openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9"
Oct 30 20:02:00.281153 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.281120    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:00.327046 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.326973    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lk8r\" (UniqueName: \"kubernetes.io/projected/9e627b8b-63bc-4af4-9743-9301f554a904-kube-api-access-5lk8r\") pod \"payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj\" (UID: \"9e627b8b-63bc-4af4-9743-9301f554a904\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj"
Oct 30 20:02:00.327046 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.327013    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e627b8b-63bc-4af4-9743-9301f554a904-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj\" (UID: \"9e627b8b-63bc-4af4-9743-9301f554a904\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj"
Oct 30 20:02:00.327248 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.327050    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44lsv\" (UniqueName: \"kubernetes.io/projected/c089c321-fd6b-4760-9f13-15ae59bd67ab-kube-api-access-44lsv\") pod \"payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9\" (UID: \"c089c321-fd6b-4760-9f13-15ae59bd67ab\") " pod="openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9"
Oct 30 20:02:00.327248 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.327086    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c089c321-fd6b-4760-9f13-15ae59bd67ab-plugins-config-volume\") pod \"payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9\" (UID: \"c089c321-fd6b-4760-9f13-15ae59bd67ab\") " pod="openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9"
Oct 30 20:02:00.327946 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.327923    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e627b8b-63bc-4af4-9743-9301f554a904-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj\" (UID: \"9e627b8b-63bc-4af4-9743-9301f554a904\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj"
Oct 30 20:02:00.328006 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.327923    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c089c321-fd6b-4760-9f13-15ae59bd67ab-plugins-config-volume\") pod \"payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9\" (UID: \"c089c321-fd6b-4760-9f13-15ae59bd67ab\") " pod="openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9"
Oct 30 20:02:00.351581 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.351549    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lk8r\" (UniqueName: \"kubernetes.io/projected/9e627b8b-63bc-4af4-9743-9301f554a904-kube-api-access-5lk8r\") pod \"payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj\" (UID: \"9e627b8b-63bc-4af4-9743-9301f554a904\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj"
Oct 30 20:02:00.351581 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.351556    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44lsv\" (UniqueName: \"kubernetes.io/projected/c089c321-fd6b-4760-9f13-15ae59bd67ab-kube-api-access-44lsv\") pod \"payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9\" (UID: \"c089c321-fd6b-4760-9f13-15ae59bd67ab\") " pod="openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9"
Oct 30 20:02:00.437707 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.437681    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj"
Oct 30 20:02:00.445738 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.445705    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9"
Oct 30 20:02:00.604282 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.604205    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:00.746492 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.746461    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj"]
Oct 30 20:02:00.749539 ip-10-0-140-248 kubenswrapper[4712]: W1030 20:02:00.749503    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e627b8b_63bc_4af4_9743_9301f554a904.slice/crio-490824c5a8b22fe92ab3ebfc994a6035f4908762e17a7072e01b50217f79df90 WatchSource:0}: Error finding container 490824c5a8b22fe92ab3ebfc994a6035f4908762e17a7072e01b50217f79df90: Status 404 returned error can't find the container with id 490824c5a8b22fe92ab3ebfc994a6035f4908762e17a7072e01b50217f79df90
Oct 30 20:02:00.761575 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:00.761545    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9"]
Oct 30 20:02:00.763079 ip-10-0-140-248 kubenswrapper[4712]: W1030 20:02:00.763048    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc089c321_fd6b_4760_9f13_15ae59bd67ab.slice/crio-5923c5c71717db22dcba886f854a47a6d7f3d37215785a01178d74348a0c38b0 WatchSource:0}: Error finding container 5923c5c71717db22dcba886f854a47a6d7f3d37215785a01178d74348a0c38b0: Status 404 returned error can't find the container with id 5923c5c71717db22dcba886f854a47a6d7f3d37215785a01178d74348a0c38b0
Oct 30 20:02:01.550120 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.550092    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"]
Oct 30 20:02:01.553577 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.553560    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"
Oct 30 20:02:01.571310 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.571282    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c8feb399-serving-cert\""
Oct 30 20:02:01.587886 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.587858    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"]
Oct 30 20:02:01.628560 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.628522    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj" event={"ID":"9e627b8b-63bc-4af4-9743-9301f554a904","Type":"ContainerStarted","Data":"d37b3e7e7a4440f2cc2e460f85998f549d78a6273f9f5f8dd4ca18cdfe1bbdf9"}
Oct 30 20:02:01.628786 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.628590    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj" event={"ID":"9e627b8b-63bc-4af4-9743-9301f554a904","Type":"ContainerStarted","Data":"490824c5a8b22fe92ab3ebfc994a6035f4908762e17a7072e01b50217f79df90"}
Oct 30 20:02:01.628786 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.628711    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj"
Oct 30 20:02:01.630151 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.630128    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9" event={"ID":"c089c321-fd6b-4760-9f13-15ae59bd67ab","Type":"ContainerStarted","Data":"32a70183609712212a45083b3e7a217a65e1c94a3c595b173ccff1698dbee7c3"}
Oct 30 20:02:01.630267 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.630164    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9" event={"ID":"c089c321-fd6b-4760-9f13-15ae59bd67ab","Type":"ContainerStarted","Data":"5923c5c71717db22dcba886f854a47a6d7f3d37215785a01178d74348a0c38b0"}
Oct 30 20:02:01.630267 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.630229    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9"
Oct 30 20:02:01.639676 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.639654    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b322601-7379-4393-85c5-db504dbc4030-openshift-service-ca\") pod \"maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp\" (UID: \"3b322601-7379-4393-85c5-db504dbc4030\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"
Oct 30 20:02:01.639783 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.639700    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b322601-7379-4393-85c5-db504dbc4030-metrics-certs\") pod \"maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp\" (UID: \"3b322601-7379-4393-85c5-db504dbc4030\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"
Oct 30 20:02:01.639783 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.639735    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b322601-7379-4393-85c5-db504dbc4030-maas-api-tls\") pod \"maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp\" (UID: \"3b322601-7379-4393-85c5-db504dbc4030\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"
Oct 30 20:02:01.639892 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.639848    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvjfc\" (UniqueName: \"kubernetes.io/projected/3b322601-7379-4393-85c5-db504dbc4030-kube-api-access-bvjfc\") pod \"maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp\" (UID: \"3b322601-7379-4393-85c5-db504dbc4030\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"
Oct 30 20:02:01.676634 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.676593    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj" podStartSLOduration=1.676582359 podStartE2EDuration="1.676582359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:01.675678069 +0000 UTC m=+2474.974940198" watchObservedRunningTime="2026-10-30 20:02:01.676582359 +0000 UTC m=+2474.975844467"
Oct 30 20:02:01.719918 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.719895    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:01.728972 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.728928    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9" podStartSLOduration=1.7289122890000002 podStartE2EDuration="1.728912289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:01.726553981 +0000 UTC m=+2475.025816110" watchObservedRunningTime="2026-10-30 20:02:01.728912289 +0000 UTC m=+2475.028174419"
Oct 30 20:02:01.740315 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.740289    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b322601-7379-4393-85c5-db504dbc4030-maas-api-tls\") pod \"maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp\" (UID: \"3b322601-7379-4393-85c5-db504dbc4030\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"
Oct 30 20:02:01.740469 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.740443    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjfc\" (UniqueName: \"kubernetes.io/projected/3b322601-7379-4393-85c5-db504dbc4030-kube-api-access-bvjfc\") pod \"maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp\" (UID: \"3b322601-7379-4393-85c5-db504dbc4030\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"
Oct 30 20:02:01.740554 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.740531    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b322601-7379-4393-85c5-db504dbc4030-openshift-service-ca\") pod \"maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp\" (UID: \"3b322601-7379-4393-85c5-db504dbc4030\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"
Oct 30 20:02:01.740713 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.740687    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b322601-7379-4393-85c5-db504dbc4030-metrics-certs\") pod \"maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp\" (UID: \"3b322601-7379-4393-85c5-db504dbc4030\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"
Oct 30 20:02:01.741278 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.741216    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b322601-7379-4393-85c5-db504dbc4030-openshift-service-ca\") pod \"maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp\" (UID: \"3b322601-7379-4393-85c5-db504dbc4030\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"
Oct 30 20:02:01.743056 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.743031    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b322601-7379-4393-85c5-db504dbc4030-maas-api-tls\") pod \"maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp\" (UID: \"3b322601-7379-4393-85c5-db504dbc4030\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"
Oct 30 20:02:01.743447 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.743428    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b322601-7379-4393-85c5-db504dbc4030-metrics-certs\") pod \"maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp\" (UID: \"3b322601-7379-4393-85c5-db504dbc4030\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"
Oct 30 20:02:01.762910 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.762885    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvjfc\" (UniqueName: \"kubernetes.io/projected/3b322601-7379-4393-85c5-db504dbc4030-kube-api-access-bvjfc\") pod \"maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp\" (UID: \"3b322601-7379-4393-85c5-db504dbc4030\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"
Oct 30 20:02:01.866959 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.866885    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"
Oct 30 20:02:01.930407 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:01.930347    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:02.217750 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:02.217724    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"]
Oct 30 20:02:02.217869 ip-10-0-140-248 kubenswrapper[4712]: W1030 20:02:02.217848    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b322601_7379_4393_85c5_db504dbc4030.slice/crio-939fd4d822060733b72fc0e6c2df5d76065365c70b1a6d3791d4cf6adc302065 WatchSource:0}: Error finding container 939fd4d822060733b72fc0e6c2df5d76065365c70b1a6d3791d4cf6adc302065: Status 404 returned error can't find the container with id 939fd4d822060733b72fc0e6c2df5d76065365c70b1a6d3791d4cf6adc302065
Oct 30 20:02:02.636395 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:02.636345    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp" event={"ID":"3b322601-7379-4393-85c5-db504dbc4030","Type":"ContainerStarted","Data":"939fd4d822060733b72fc0e6c2df5d76065365c70b1a6d3791d4cf6adc302065"}
Oct 30 20:02:03.615735 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:03.615697    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:03.642322 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:03.642289    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp" event={"ID":"3b322601-7379-4393-85c5-db504dbc4030","Type":"ContainerStarted","Data":"9f5939f40f7a4405f9178b53f254e94e05341d47ccae0a195df3f843885277f1"}
Oct 30 20:02:03.642713 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:03.642393    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"
Oct 30 20:02:03.688092 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:03.688047    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp" podStartSLOduration=2.327890995 podStartE2EDuration="2.688035024s" totalImagesPullingTime="360.144029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:03.686114935 +0000 UTC m=+2476.985377063" watchObservedRunningTime="2026-10-30 20:02:03.688035024 +0000 UTC m=+2476.987297157"
Oct 30 20:02:05.187410 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:05.187380    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:06.372587 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:02:06.372550    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 20:02:06.793756 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:06.793653    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:08.282232 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:08.282200    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:09.652281 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:09.652254    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"
Oct 30 20:02:09.738468 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:09.736822    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:11.334110 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:11.334009    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:12.640539 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:12.640503    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj"
Oct 30 20:02:12.641045 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:12.640592    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9"
Oct 30 20:02:12.788039 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:12.788005    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:14.487264 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:14.487218    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:14.696408 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:14.696374    4712 generic.go:356] "Generic (PLEG): container finished" podID="a9b26d2c-afba-44d7-8323-555420014ee1" containerID="b4f790a78b27ff3b1a747d272ebdae8334c937a203a5f60cd2089e94cdd1125d" exitCode=0
Oct 30 20:02:14.696408 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:14.696393    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2" event={"ID":"a9b26d2c-afba-44d7-8323-555420014ee1","Type":"ContainerDied","Data":"b4f790a78b27ff3b1a747d272ebdae8334c937a203a5f60cd2089e94cdd1125d"}
Oct 30 20:02:14.696787 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:14.696772    4712 scope.go:122] "RemoveContainer" containerID="b4f790a78b27ff3b1a747d272ebdae8334c937a203a5f60cd2089e94cdd1125d"
Oct 30 20:02:14.698024 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:14.697994    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:02:14.698105 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:14.698068    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:02:14.698105 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:14.698101    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:02:15.702414 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:15.702377    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2" event={"ID":"a9b26d2c-afba-44d7-8323-555420014ee1","Type":"ContainerStarted","Data":"550c5a87e1a547a7746a5f2fefa6c104ace95ce8b32bfe7b0f4b493af742b89f"}
Oct 30 20:02:15.988696 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:15.988615    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:16.525386 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:16.524702    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:02:16.525386 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:16.524818    4712 patch_prober.go:29] interesting pod/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:02:16.525386 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:16.524862    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2" podUID="a9b26d2c-afba-44d7-8323-555420014ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:02:17.526386 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:17.525704    4712 patch_prober.go:29] interesting pod/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:02:17.526386 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:17.525794    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2" podUID="a9b26d2c-afba-44d7-8323-555420014ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:02:17.697420 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:17.697386    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:18.527175 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:18.526696    4712 patch_prober.go:29] interesting pod/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:02:18.527175 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:18.526783    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2" podUID="a9b26d2c-afba-44d7-8323-555420014ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:02:19.387560 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:19.387516    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:19.525745 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:19.525698    4712 patch_prober.go:29] interesting pod/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:02:19.526027 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:19.525997    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2" podUID="a9b26d2c-afba-44d7-8323-555420014ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:02:20.522213 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:20.522162    4712 patch_prober.go:29] interesting pod/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:02:20.522763 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:20.522229    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2" podUID="a9b26d2c-afba-44d7-8323-555420014ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:02:21.018515 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:21.018461    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:21.525166 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:21.525134    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:02:21.525657 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:21.525525    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:02:21.526205 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:21.526184    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:02:22.568109 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:22.568077    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:24.178922 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:24.178885    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:25.690799 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:25.690763    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:27.174066 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:27.174037    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:28.739198 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:28.739165    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:28.857177 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:28.857149    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9"]
Oct 30 20:02:28.873668 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:28.873644    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj"]
Oct 30 20:02:29.019786 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:29.019711    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:29.179778 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:29.179747    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:29.967946 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:29.967913    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"]
Oct 30 20:02:29.968468 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:29.968100    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp" podUID="3b322601-7379-4393-85c5-db504dbc4030" containerName="maas-api" containerID="cri-o://9f5939f40f7a4405f9178b53f254e94e05341d47ccae0a195df3f843885277f1" gracePeriod=30
Oct 30 20:02:30.104524 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.104495    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:30.212511 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.212486    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"
Oct 30 20:02:30.243326 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.243257    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b322601-7379-4393-85c5-db504dbc4030-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b322601-7379-4393-85c5-db504dbc4030-metrics-certs\") pod \"3b322601-7379-4393-85c5-db504dbc4030\" (UID: \"3b322601-7379-4393-85c5-db504dbc4030\") "
Oct 30 20:02:30.243326 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.243303    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b322601-7379-4393-85c5-db504dbc4030-kube-api-access-bvjfc\" (UniqueName: \"kubernetes.io/projected/3b322601-7379-4393-85c5-db504dbc4030-kube-api-access-bvjfc\") pod \"3b322601-7379-4393-85c5-db504dbc4030\" (UID: \"3b322601-7379-4393-85c5-db504dbc4030\") "
Oct 30 20:02:30.243523 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.243355    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b322601-7379-4393-85c5-db504dbc4030-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b322601-7379-4393-85c5-db504dbc4030-openshift-service-ca\") pod \"3b322601-7379-4393-85c5-db504dbc4030\" (UID: \"3b322601-7379-4393-85c5-db504dbc4030\") "
Oct 30 20:02:30.243523 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.243437    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b322601-7379-4393-85c5-db504dbc4030-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b322601-7379-4393-85c5-db504dbc4030-maas-api-tls\") pod \"3b322601-7379-4393-85c5-db504dbc4030\" (UID: \"3b322601-7379-4393-85c5-db504dbc4030\") "
Oct 30 20:02:30.243809 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.243782    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b322601-7379-4393-85c5-db504dbc4030-openshift-service-ca" pod "3b322601-7379-4393-85c5-db504dbc4030" (UID: "3b322601-7379-4393-85c5-db504dbc4030"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:30.245410 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.245335    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b322601-7379-4393-85c5-db504dbc4030-metrics-certs" pod "3b322601-7379-4393-85c5-db504dbc4030" (UID: "3b322601-7379-4393-85c5-db504dbc4030"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:30.245531 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.245445    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b322601-7379-4393-85c5-db504dbc4030-maas-api-tls" pod "3b322601-7379-4393-85c5-db504dbc4030" (UID: "3b322601-7379-4393-85c5-db504dbc4030"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:30.245531 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.245456    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b322601-7379-4393-85c5-db504dbc4030-kube-api-access-bvjfc" pod "3b322601-7379-4393-85c5-db504dbc4030" (UID: "3b322601-7379-4393-85c5-db504dbc4030"). InnerVolumeSpecName "kube-api-access-bvjfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:30.344061 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.344037    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvjfc\" (UniqueName: \"kubernetes.io/projected/3b322601-7379-4393-85c5-db504dbc4030-kube-api-access-bvjfc\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:02:30.344061 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.344061    4712 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b322601-7379-4393-85c5-db504dbc4030-openshift-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:02:30.344246 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.344071    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b322601-7379-4393-85c5-db504dbc4030-maas-api-tls\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:02:30.344246 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.344081    4712 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b322601-7379-4393-85c5-db504dbc4030-metrics-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:02:30.565577 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.565503    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:30.712427 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.712394    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:30.791793 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.791759    4712 generic.go:356] "Generic (PLEG): container finished" podID="3b322601-7379-4393-85c5-db504dbc4030" containerID="9f5939f40f7a4405f9178b53f254e94e05341d47ccae0a195df3f843885277f1" exitCode=0
Oct 30 20:02:30.791992 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.791804    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"
Oct 30 20:02:30.791992 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.791841    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp" event={"ID":"3b322601-7379-4393-85c5-db504dbc4030","Type":"ContainerDied","Data":"9f5939f40f7a4405f9178b53f254e94e05341d47ccae0a195df3f843885277f1"}
Oct 30 20:02:30.791992 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.791894    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp" event={"ID":"3b322601-7379-4393-85c5-db504dbc4030","Type":"ContainerDied","Data":"939fd4d822060733b72fc0e6c2df5d76065365c70b1a6d3791d4cf6adc302065"}
Oct 30 20:02:30.791992 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.791918    4712 scope.go:122] "RemoveContainer" containerID="9f5939f40f7a4405f9178b53f254e94e05341d47ccae0a195df3f843885277f1"
Oct 30 20:02:30.801540 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.801522    4712 scope.go:122] "RemoveContainer" containerID="9f5939f40f7a4405f9178b53f254e94e05341d47ccae0a195df3f843885277f1"
Oct 30 20:02:30.804651 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:02:30.804624    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5939f40f7a4405f9178b53f254e94e05341d47ccae0a195df3f843885277f1\": container with ID starting with 9f5939f40f7a4405f9178b53f254e94e05341d47ccae0a195df3f843885277f1 not found: ID does not exist" containerID="9f5939f40f7a4405f9178b53f254e94e05341d47ccae0a195df3f843885277f1"
Oct 30 20:02:30.804751 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.804660    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5939f40f7a4405f9178b53f254e94e05341d47ccae0a195df3f843885277f1"} err="failed to get container status \"9f5939f40f7a4405f9178b53f254e94e05341d47ccae0a195df3f843885277f1\": rpc error: code = NotFound desc = could not find container \"9f5939f40f7a4405f9178b53f254e94e05341d47ccae0a195df3f843885277f1\": container with ID starting with 9f5939f40f7a4405f9178b53f254e94e05341d47ccae0a195df3f843885277f1 not found: ID does not exist"
Oct 30 20:02:30.841217 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.841163    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"]
Oct 30 20:02:30.853161 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:30.853141    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c8feb399-5956dfbff4-dwbcp"]
Oct 30 20:02:32.851384 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:32.851320    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b322601-7379-4393-85c5-db504dbc4030" path="/var/lib/kubelet/pods/3b322601-7379-4393-85c5-db504dbc4030/volumes"
Oct 30 20:02:33.858377 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:33.858311    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9" podUID="c089c321-fd6b-4760-9f13-15ae59bd67ab" containerName="payload-processing" containerID="cri-o://32a70183609712212a45083b3e7a217a65e1c94a3c595b173ccff1698dbee7c3" gracePeriod=25
Oct 30 20:02:33.873922 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:33.873895    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj" podUID="9e627b8b-63bc-4af4-9743-9301f554a904" containerName="payload-pre-processing" containerID="cri-o://d37b3e7e7a4440f2cc2e460f85998f549d78a6273f9f5f8dd4ca18cdfe1bbdf9" gracePeriod=25
Oct 30 20:02:34.145163 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.145138    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9"
Oct 30 20:02:34.148006 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.147984    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj"
Oct 30 20:02:34.175637 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.175614    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e627b8b-63bc-4af4-9743-9301f554a904-kube-api-access-5lk8r\" (UniqueName: \"kubernetes.io/projected/9e627b8b-63bc-4af4-9743-9301f554a904-kube-api-access-5lk8r\") pod \"9e627b8b-63bc-4af4-9743-9301f554a904\" (UID: \"9e627b8b-63bc-4af4-9743-9301f554a904\") "
Oct 30 20:02:34.175730 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.175645    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e627b8b-63bc-4af4-9743-9301f554a904-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e627b8b-63bc-4af4-9743-9301f554a904-plugins-config-volume\") pod \"9e627b8b-63bc-4af4-9743-9301f554a904\" (UID: \"9e627b8b-63bc-4af4-9743-9301f554a904\") "
Oct 30 20:02:34.175730 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.175697    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c089c321-fd6b-4760-9f13-15ae59bd67ab-kube-api-access-44lsv\" (UniqueName: \"kubernetes.io/projected/c089c321-fd6b-4760-9f13-15ae59bd67ab-kube-api-access-44lsv\") pod \"c089c321-fd6b-4760-9f13-15ae59bd67ab\" (UID: \"c089c321-fd6b-4760-9f13-15ae59bd67ab\") "
Oct 30 20:02:34.175844 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.175735    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c089c321-fd6b-4760-9f13-15ae59bd67ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c089c321-fd6b-4760-9f13-15ae59bd67ab-plugins-config-volume\") pod \"c089c321-fd6b-4760-9f13-15ae59bd67ab\" (UID: \"c089c321-fd6b-4760-9f13-15ae59bd67ab\") "
Oct 30 20:02:34.176076 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.176055    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e627b8b-63bc-4af4-9743-9301f554a904-plugins-config-volume" pod "9e627b8b-63bc-4af4-9743-9301f554a904" (UID: "9e627b8b-63bc-4af4-9743-9301f554a904"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:34.176144 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.176092    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c089c321-fd6b-4760-9f13-15ae59bd67ab-plugins-config-volume" pod "c089c321-fd6b-4760-9f13-15ae59bd67ab" (UID: "c089c321-fd6b-4760-9f13-15ae59bd67ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:34.177592 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.177574    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e627b8b-63bc-4af4-9743-9301f554a904-kube-api-access-5lk8r" pod "9e627b8b-63bc-4af4-9743-9301f554a904" (UID: "9e627b8b-63bc-4af4-9743-9301f554a904"). InnerVolumeSpecName "kube-api-access-5lk8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:34.177655 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.177630    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c089c321-fd6b-4760-9f13-15ae59bd67ab-kube-api-access-44lsv" pod "c089c321-fd6b-4760-9f13-15ae59bd67ab" (UID: "c089c321-fd6b-4760-9f13-15ae59bd67ab"). InnerVolumeSpecName "kube-api-access-44lsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:34.276602 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.276574    4712 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c089c321-fd6b-4760-9f13-15ae59bd67ab-plugins-config-volume\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:02:34.276602 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.276598    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lk8r\" (UniqueName: \"kubernetes.io/projected/9e627b8b-63bc-4af4-9743-9301f554a904-kube-api-access-5lk8r\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:02:34.276602 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.276608    4712 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e627b8b-63bc-4af4-9743-9301f554a904-plugins-config-volume\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:02:34.276791 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.276616    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44lsv\" (UniqueName: \"kubernetes.io/projected/c089c321-fd6b-4760-9f13-15ae59bd67ab-kube-api-access-44lsv\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:02:34.748025 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.747990    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"]
Oct 30 20:02:34.748501 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.748261    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2" podUID="a9b26d2c-afba-44d7-8323-555420014ee1" containerName="istio-proxy" containerID="cri-o://550c5a87e1a547a7746a5f2fefa6c104ace95ce8b32bfe7b0f4b493af742b89f" gracePeriod=30
Oct 30 20:02:34.809220 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.809185    4712 generic.go:356] "Generic (PLEG): container finished" podID="9e627b8b-63bc-4af4-9743-9301f554a904" containerID="d37b3e7e7a4440f2cc2e460f85998f549d78a6273f9f5f8dd4ca18cdfe1bbdf9" exitCode=0
Oct 30 20:02:34.809409 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.809228    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj"
Oct 30 20:02:34.809409 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.809269    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj" event={"ID":"9e627b8b-63bc-4af4-9743-9301f554a904","Type":"ContainerDied","Data":"d37b3e7e7a4440f2cc2e460f85998f549d78a6273f9f5f8dd4ca18cdfe1bbdf9"}
Oct 30 20:02:34.809409 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.809327    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj" event={"ID":"9e627b8b-63bc-4af4-9743-9301f554a904","Type":"ContainerDied","Data":"490824c5a8b22fe92ab3ebfc994a6035f4908762e17a7072e01b50217f79df90"}
Oct 30 20:02:34.809409 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.809356    4712 scope.go:122] "RemoveContainer" containerID="d37b3e7e7a4440f2cc2e460f85998f549d78a6273f9f5f8dd4ca18cdfe1bbdf9"
Oct 30 20:02:34.810662 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.810642    4712 generic.go:356] "Generic (PLEG): container finished" podID="c089c321-fd6b-4760-9f13-15ae59bd67ab" containerID="32a70183609712212a45083b3e7a217a65e1c94a3c595b173ccff1698dbee7c3" exitCode=0
Oct 30 20:02:34.810756 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.810672    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9" event={"ID":"c089c321-fd6b-4760-9f13-15ae59bd67ab","Type":"ContainerDied","Data":"32a70183609712212a45083b3e7a217a65e1c94a3c595b173ccff1698dbee7c3"}
Oct 30 20:02:34.810756 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.810692    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9"
Oct 30 20:02:34.810756 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.810723    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9" event={"ID":"c089c321-fd6b-4760-9f13-15ae59bd67ab","Type":"ContainerDied","Data":"5923c5c71717db22dcba886f854a47a6d7f3d37215785a01178d74348a0c38b0"}
Oct 30 20:02:34.817669 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.817649    4712 scope.go:122] "RemoveContainer" containerID="d37b3e7e7a4440f2cc2e460f85998f549d78a6273f9f5f8dd4ca18cdfe1bbdf9"
Oct 30 20:02:34.817911 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:02:34.817895    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37b3e7e7a4440f2cc2e460f85998f549d78a6273f9f5f8dd4ca18cdfe1bbdf9\": container with ID starting with d37b3e7e7a4440f2cc2e460f85998f549d78a6273f9f5f8dd4ca18cdfe1bbdf9 not found: ID does not exist" containerID="d37b3e7e7a4440f2cc2e460f85998f549d78a6273f9f5f8dd4ca18cdfe1bbdf9"
Oct 30 20:02:34.817963 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.817918    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37b3e7e7a4440f2cc2e460f85998f549d78a6273f9f5f8dd4ca18cdfe1bbdf9"} err="failed to get container status \"d37b3e7e7a4440f2cc2e460f85998f549d78a6273f9f5f8dd4ca18cdfe1bbdf9\": rpc error: code = NotFound desc = could not find container \"d37b3e7e7a4440f2cc2e460f85998f549d78a6273f9f5f8dd4ca18cdfe1bbdf9\": container with ID starting with d37b3e7e7a4440f2cc2e460f85998f549d78a6273f9f5f8dd4ca18cdfe1bbdf9 not found: ID does not exist"
Oct 30 20:02:34.817963 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.817935    4712 scope.go:122] "RemoveContainer" containerID="32a70183609712212a45083b3e7a217a65e1c94a3c595b173ccff1698dbee7c3"
Oct 30 20:02:34.824348 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.824307    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:34.825859 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.825843    4712 scope.go:122] "RemoveContainer" containerID="32a70183609712212a45083b3e7a217a65e1c94a3c595b173ccff1698dbee7c3"
Oct 30 20:02:34.826105 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:02:34.826088    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a70183609712212a45083b3e7a217a65e1c94a3c595b173ccff1698dbee7c3\": container with ID starting with 32a70183609712212a45083b3e7a217a65e1c94a3c595b173ccff1698dbee7c3 not found: ID does not exist" containerID="32a70183609712212a45083b3e7a217a65e1c94a3c595b173ccff1698dbee7c3"
Oct 30 20:02:34.826145 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.826110    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a70183609712212a45083b3e7a217a65e1c94a3c595b173ccff1698dbee7c3"} err="failed to get container status \"32a70183609712212a45083b3e7a217a65e1c94a3c595b173ccff1698dbee7c3\": rpc error: code = NotFound desc = could not find container \"32a70183609712212a45083b3e7a217a65e1c94a3c595b173ccff1698dbee7c3\": container with ID starting with 32a70183609712212a45083b3e7a217a65e1c94a3c595b173ccff1698dbee7c3 not found: ID does not exist"
Oct 30 20:02:34.859634 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.859598    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9"]
Oct 30 20:02:34.863464 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.863434    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c8feb399-8bf477cb4-lpjv9"]
Oct 30 20:02:34.895582 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.895556    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj"]
Oct 30 20:02:34.901327 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:34.901304    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c8feb399-679d48b6dc-lxlgj"]
Oct 30 20:02:35.060184 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:35.060118    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:36.133348 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.133317    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:36.372374 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:02:36.372323    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 20:02:36.509057 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.508975    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"]
Oct 30 20:02:36.509691 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.509665    4712 cpu_manager.go:414] "containerMap: removing container" podUID="c089c321-fd6b-4760-9f13-15ae59bd67ab" containerName="payload-processing"
Oct 30 20:02:36.509754 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.509696    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c089c321-fd6b-4760-9f13-15ae59bd67ab" containerName="payload-processing"
Oct 30 20:02:36.509754 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.509740    4712 cpu_manager.go:414] "containerMap: removing container" podUID="9e627b8b-63bc-4af4-9743-9301f554a904" containerName="payload-pre-processing"
Oct 30 20:02:36.509754 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.509749    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e627b8b-63bc-4af4-9743-9301f554a904" containerName="payload-pre-processing"
Oct 30 20:02:36.509853 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.509774    4712 cpu_manager.go:414] "containerMap: removing container" podUID="3b322601-7379-4393-85c5-db504dbc4030" containerName="maas-api"
Oct 30 20:02:36.509853 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.509784    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b322601-7379-4393-85c5-db504dbc4030" containerName="maas-api"
Oct 30 20:02:36.509916 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.509895    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="c089c321-fd6b-4760-9f13-15ae59bd67ab" containerName="payload-processing"
Oct 30 20:02:36.509948 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.509917    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e627b8b-63bc-4af4-9743-9301f554a904" containerName="payload-pre-processing"
Oct 30 20:02:36.509948 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.509930    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b322601-7379-4393-85c5-db504dbc4030" containerName="maas-api"
Oct 30 20:02:36.514994 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.514964    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.528441 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.528415    4712 patch_prober.go:29] interesting pod/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:02:36.528562 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.528464    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2" podUID="a9b26d2c-afba-44d7-8323-555420014ee1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:02:36.529913 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.529896    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-31a039-openshift-default-dockercfg-46p5d\""
Oct 30 20:02:36.543788 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.543766    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"]
Oct 30 20:02:36.595451 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.595421    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-credential-socket\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.595553 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.595468    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d7df99-9688-4e01-b42c-30bc86b89295-istiod-ca-cert\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.595553 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.595542    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-workload-certs\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.595638 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.595570    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-workload-socket\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.595638 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.595588    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-envoy\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.595709 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.595645    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-data\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.595709 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.595674    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-podinfo\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.595709 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.595700    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z4v5\" (UniqueName: \"kubernetes.io/projected/f3d7df99-9688-4e01-b42c-30bc86b89295-kube-api-access-7z4v5\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.595808 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.595797    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-token\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.608622 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.608597    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:36.696310 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.696281    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-data\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.696513 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.696347    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-podinfo\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.696513 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.696399    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4v5\" (UniqueName: \"kubernetes.io/projected/f3d7df99-9688-4e01-b42c-30bc86b89295-kube-api-access-7z4v5\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.696513 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.696444    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-token\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.696513 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.696478    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-credential-socket\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.696724 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.696520    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d7df99-9688-4e01-b42c-30bc86b89295-istiod-ca-cert\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.696724 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.696577    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-workload-certs\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.696724 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.696624    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-workload-socket\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.696724 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.696654    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-envoy\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.696940 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.696760    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-data\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.697640 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.697613    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d7df99-9688-4e01-b42c-30bc86b89295-istiod-ca-cert\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.698400 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.698116    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-workload-socket\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.698400 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.698345    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-credential-socket\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.698743 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.698719    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-workload-certs\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.699552 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.699519    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-envoy\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.700783 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.700758    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-podinfo\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.716851 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.716809    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-token\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.717052 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.717028    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z4v5\" (UniqueName: \"kubernetes.io/projected/f3d7df99-9688-4e01-b42c-30bc86b89295-kube-api-access-7z4v5\") pod \"e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") " pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.797296 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.797225    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:36.831999 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.831976    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:36.852839 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.852808    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e627b8b-63bc-4af4-9743-9301f554a904" path="/var/lib/kubelet/pods/9e627b8b-63bc-4af4-9743-9301f554a904/volumes"
Oct 30 20:02:36.853396 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:36.853356    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c089c321-fd6b-4760-9f13-15ae59bd67ab" path="/var/lib/kubelet/pods/c089c321-fd6b-4760-9f13-15ae59bd67ab/volumes"
Oct 30 20:02:37.140024 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:37.139993    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"]
Oct 30 20:02:37.140448 ip-10-0-140-248 kubenswrapper[4712]: W1030 20:02:37.140423    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d7df99_9688_4e01_b42c_30bc86b89295.slice/crio-1a6a48fd3fcd42c6ce6d29dfd3f7fb3d474c88311cc4c0ac21afabac15387218 WatchSource:0}: Error finding container 1a6a48fd3fcd42c6ce6d29dfd3f7fb3d474c88311cc4c0ac21afabac15387218: Status 404 returned error can't find the container with id 1a6a48fd3fcd42c6ce6d29dfd3f7fb3d474c88311cc4c0ac21afabac15387218
Oct 30 20:02:37.142526 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:37.142496    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:02:37.142586 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:37.142556    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:02:37.142621 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:37.142585    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:02:37.828181 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:37.828146    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4" event={"ID":"f3d7df99-9688-4e01-b42c-30bc86b89295","Type":"ContainerStarted","Data":"7f1ae2c2220bb304a6fbbd7589b656d4cb85a21443c91ca2056c8f3069a9fed7"}
Oct 30 20:02:37.828346 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:37.828203    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4" event={"ID":"f3d7df99-9688-4e01-b42c-30bc86b89295","Type":"ContainerStarted","Data":"1a6a48fd3fcd42c6ce6d29dfd3f7fb3d474c88311cc4c0ac21afabac15387218"}
Oct 30 20:02:37.832479 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:37.832460    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:38.837332 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:38.837301    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:38.901322 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:38.901270    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4" podStartSLOduration=2.901256869 podStartE2EDuration="2.901256869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:37.896124042 +0000 UTC m=+2511.195386171" watchObservedRunningTime="2026-10-30 20:02:38.901256869 +0000 UTC m=+2512.200519053"
Oct 30 20:02:39.836462 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:39.836437    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:39.837273 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:39.837251    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:40.005888 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.005863    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:02:40.125621 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.125591    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-istio-envoy\") pod \"a9b26d2c-afba-44d7-8323-555420014ee1\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") "
Oct 30 20:02:40.125797 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.125639    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-istio-data\") pod \"a9b26d2c-afba-44d7-8323-555420014ee1\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") "
Oct 30 20:02:40.125797 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.125686    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-credential-socket\") pod \"a9b26d2c-afba-44d7-8323-555420014ee1\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") "
Oct 30 20:02:40.125797 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.125711    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-workload-socket\") pod \"a9b26d2c-afba-44d7-8323-555420014ee1\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") "
Oct 30 20:02:40.125797 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.125744    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b26d2c-afba-44d7-8323-555420014ee1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b26d2c-afba-44d7-8323-555420014ee1-istiod-ca-cert\") pod \"a9b26d2c-afba-44d7-8323-555420014ee1\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") "
Oct 30 20:02:40.125797 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.125792    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-workload-certs\") pod \"a9b26d2c-afba-44d7-8323-555420014ee1\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") "
Oct 30 20:02:40.126033 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.125850    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9b26d2c-afba-44d7-8323-555420014ee1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b26d2c-afba-44d7-8323-555420014ee1-istio-podinfo\") pod \"a9b26d2c-afba-44d7-8323-555420014ee1\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") "
Oct 30 20:02:40.126033 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.125899    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b26d2c-afba-44d7-8323-555420014ee1-kube-api-access-7jxfw\" (UniqueName: \"kubernetes.io/projected/a9b26d2c-afba-44d7-8323-555420014ee1-kube-api-access-7jxfw\") pod \"a9b26d2c-afba-44d7-8323-555420014ee1\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") "
Oct 30 20:02:40.126033 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.125953    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b26d2c-afba-44d7-8323-555420014ee1-istio-token\" (UniqueName: \"kubernetes.io/projected/a9b26d2c-afba-44d7-8323-555420014ee1-istio-token\") pod \"a9b26d2c-afba-44d7-8323-555420014ee1\" (UID: \"a9b26d2c-afba-44d7-8323-555420014ee1\") "
Oct 30 20:02:40.126033 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.125955    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-istio-data" pod "a9b26d2c-afba-44d7-8323-555420014ee1" (UID: "a9b26d2c-afba-44d7-8323-555420014ee1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:40.126243 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.126042    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-credential-socket" pod "a9b26d2c-afba-44d7-8323-555420014ee1" (UID: "a9b26d2c-afba-44d7-8323-555420014ee1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:40.126243 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.126055    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-workload-socket" pod "a9b26d2c-afba-44d7-8323-555420014ee1" (UID: "a9b26d2c-afba-44d7-8323-555420014ee1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:40.126243 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.126146    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-workload-certs" pod "a9b26d2c-afba-44d7-8323-555420014ee1" (UID: "a9b26d2c-afba-44d7-8323-555420014ee1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:40.126421 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.126391    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:02:40.126477 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.126416    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:02:40.126477 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.126423    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b26d2c-afba-44d7-8323-555420014ee1-istiod-ca-cert" pod "a9b26d2c-afba-44d7-8323-555420014ee1" (UID: "a9b26d2c-afba-44d7-8323-555420014ee1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:40.126477 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.126434    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:02:40.126477 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.126463    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:02:40.128101 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.128075    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9b26d2c-afba-44d7-8323-555420014ee1-istio-podinfo" pod "a9b26d2c-afba-44d7-8323-555420014ee1" (UID: "a9b26d2c-afba-44d7-8323-555420014ee1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:40.128220 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.128102    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b26d2c-afba-44d7-8323-555420014ee1-istio-token" pod "a9b26d2c-afba-44d7-8323-555420014ee1" (UID: "a9b26d2c-afba-44d7-8323-555420014ee1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:40.128391 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.128352    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b26d2c-afba-44d7-8323-555420014ee1-kube-api-access-7jxfw" pod "a9b26d2c-afba-44d7-8323-555420014ee1" (UID: "a9b26d2c-afba-44d7-8323-555420014ee1"). InnerVolumeSpecName "kube-api-access-7jxfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:40.128472 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.128357    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-istio-envoy" pod "a9b26d2c-afba-44d7-8323-555420014ee1" (UID: "a9b26d2c-afba-44d7-8323-555420014ee1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:40.226992 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.226967    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jxfw\" (UniqueName: \"kubernetes.io/projected/a9b26d2c-afba-44d7-8323-555420014ee1-kube-api-access-7jxfw\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:02:40.226992 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.226990    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b26d2c-afba-44d7-8323-555420014ee1-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:02:40.227140 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.227000    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b26d2c-afba-44d7-8323-555420014ee1-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:02:40.227140 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.227009    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b26d2c-afba-44d7-8323-555420014ee1-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:02:40.227140 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.227016    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b26d2c-afba-44d7-8323-555420014ee1-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:02:40.341325 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.341300    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:40.511448 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.511386    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:40.842617 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.842583    4712 generic.go:356] "Generic (PLEG): container finished" podID="a9b26d2c-afba-44d7-8323-555420014ee1" containerID="550c5a87e1a547a7746a5f2fefa6c104ace95ce8b32bfe7b0f4b493af742b89f" exitCode=0
Oct 30 20:02:40.842750 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.842633    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"
Oct 30 20:02:40.842750 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.842625    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2" event={"ID":"a9b26d2c-afba-44d7-8323-555420014ee1","Type":"ContainerDied","Data":"550c5a87e1a547a7746a5f2fefa6c104ace95ce8b32bfe7b0f4b493af742b89f"}
Oct 30 20:02:40.842890 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.842767    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2" event={"ID":"a9b26d2c-afba-44d7-8323-555420014ee1","Type":"ContainerDied","Data":"7a20d67f7c1541dbd8e417b7a1725e922528c479b8d376b61390d77464be30fd"}
Oct 30 20:02:40.842890 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.842800    4712 scope.go:122] "RemoveContainer" containerID="550c5a87e1a547a7746a5f2fefa6c104ace95ce8b32bfe7b0f4b493af742b89f"
Oct 30 20:02:40.852048 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.852035    4712 scope.go:122] "RemoveContainer" containerID="b4f790a78b27ff3b1a747d272ebdae8334c937a203a5f60cd2089e94cdd1125d"
Oct 30 20:02:40.859763 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.859747    4712 scope.go:122] "RemoveContainer" containerID="550c5a87e1a547a7746a5f2fefa6c104ace95ce8b32bfe7b0f4b493af742b89f"
Oct 30 20:02:40.860046 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:02:40.860026    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"550c5a87e1a547a7746a5f2fefa6c104ace95ce8b32bfe7b0f4b493af742b89f\": container with ID starting with 550c5a87e1a547a7746a5f2fefa6c104ace95ce8b32bfe7b0f4b493af742b89f not found: ID does not exist" containerID="550c5a87e1a547a7746a5f2fefa6c104ace95ce8b32bfe7b0f4b493af742b89f"
Oct 30 20:02:40.860102 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.860054    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"550c5a87e1a547a7746a5f2fefa6c104ace95ce8b32bfe7b0f4b493af742b89f"} err="failed to get container status \"550c5a87e1a547a7746a5f2fefa6c104ace95ce8b32bfe7b0f4b493af742b89f\": rpc error: code = NotFound desc = could not find container \"550c5a87e1a547a7746a5f2fefa6c104ace95ce8b32bfe7b0f4b493af742b89f\": container with ID starting with 550c5a87e1a547a7746a5f2fefa6c104ace95ce8b32bfe7b0f4b493af742b89f not found: ID does not exist"
Oct 30 20:02:40.860102 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.860072    4712 scope.go:122] "RemoveContainer" containerID="b4f790a78b27ff3b1a747d272ebdae8334c937a203a5f60cd2089e94cdd1125d"
Oct 30 20:02:40.860315 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:02:40.860299    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4f790a78b27ff3b1a747d272ebdae8334c937a203a5f60cd2089e94cdd1125d\": container with ID starting with b4f790a78b27ff3b1a747d272ebdae8334c937a203a5f60cd2089e94cdd1125d not found: ID does not exist" containerID="b4f790a78b27ff3b1a747d272ebdae8334c937a203a5f60cd2089e94cdd1125d"
Oct 30 20:02:40.860384 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.860321    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4f790a78b27ff3b1a747d272ebdae8334c937a203a5f60cd2089e94cdd1125d"} err="failed to get container status \"b4f790a78b27ff3b1a747d272ebdae8334c937a203a5f60cd2089e94cdd1125d\": rpc error: code = NotFound desc = could not find container \"b4f790a78b27ff3b1a747d272ebdae8334c937a203a5f60cd2089e94cdd1125d\": container with ID starting with b4f790a78b27ff3b1a747d272ebdae8334c937a203a5f60cd2089e94cdd1125d not found: ID does not exist"
Oct 30 20:02:40.907386 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.907275    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"]
Oct 30 20:02:40.917877 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:40.917857    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c8feb399-openshift-default-5fc9cc9bdc-gvzg2"]
Oct 30 20:02:41.720061 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.720024    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"]
Oct 30 20:02:41.720460 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.720413    4712 cpu_manager.go:414] "containerMap: removing container" podUID="a9b26d2c-afba-44d7-8323-555420014ee1" containerName="istio-proxy"
Oct 30 20:02:41.720460 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.720425    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b26d2c-afba-44d7-8323-555420014ee1" containerName="istio-proxy"
Oct 30 20:02:41.720460 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.720436    4712 cpu_manager.go:414] "containerMap: removing container" podUID="a9b26d2c-afba-44d7-8323-555420014ee1" containerName="istio-proxy"
Oct 30 20:02:41.720460 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.720441    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b26d2c-afba-44d7-8323-555420014ee1" containerName="istio-proxy"
Oct 30 20:02:41.720592 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.720517    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b26d2c-afba-44d7-8323-555420014ee1" containerName="istio-proxy"
Oct 30 20:02:41.720592 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.720524    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b26d2c-afba-44d7-8323-555420014ee1" containerName="istio-proxy"
Oct 30 20:02:41.724898 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.724878    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"
Oct 30 20:02:41.751066 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.751044    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-31a039-serving-cert\""
Oct 30 20:02:41.762616 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.762593    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"]
Oct 30 20:02:41.842220 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.842191    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33874386-eb7c-4ca7-82ff-9452d51648b4-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55\" (UID: \"33874386-eb7c-4ca7-82ff-9452d51648b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"
Oct 30 20:02:41.842371 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.842240    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfgm8\" (UniqueName: \"kubernetes.io/projected/33874386-eb7c-4ca7-82ff-9452d51648b4-kube-api-access-gfgm8\") pod \"maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55\" (UID: \"33874386-eb7c-4ca7-82ff-9452d51648b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"
Oct 30 20:02:41.842371 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.842259    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33874386-eb7c-4ca7-82ff-9452d51648b4-metrics-certs\") pod \"maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55\" (UID: \"33874386-eb7c-4ca7-82ff-9452d51648b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"
Oct 30 20:02:41.842465 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.842422    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33874386-eb7c-4ca7-82ff-9452d51648b4-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55\" (UID: \"33874386-eb7c-4ca7-82ff-9452d51648b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"
Oct 30 20:02:41.897296 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.897269    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:41.943407 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.943348    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33874386-eb7c-4ca7-82ff-9452d51648b4-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55\" (UID: \"33874386-eb7c-4ca7-82ff-9452d51648b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"
Oct 30 20:02:41.943620 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.943530    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfgm8\" (UniqueName: \"kubernetes.io/projected/33874386-eb7c-4ca7-82ff-9452d51648b4-kube-api-access-gfgm8\") pod \"maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55\" (UID: \"33874386-eb7c-4ca7-82ff-9452d51648b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"
Oct 30 20:02:41.943620 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.943555    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33874386-eb7c-4ca7-82ff-9452d51648b4-metrics-certs\") pod \"maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55\" (UID: \"33874386-eb7c-4ca7-82ff-9452d51648b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"
Oct 30 20:02:41.943921 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.943896    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33874386-eb7c-4ca7-82ff-9452d51648b4-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55\" (UID: \"33874386-eb7c-4ca7-82ff-9452d51648b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"
Oct 30 20:02:41.944567 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.944545    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33874386-eb7c-4ca7-82ff-9452d51648b4-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55\" (UID: \"33874386-eb7c-4ca7-82ff-9452d51648b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"
Oct 30 20:02:41.946276 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.946250    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33874386-eb7c-4ca7-82ff-9452d51648b4-metrics-certs\") pod \"maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55\" (UID: \"33874386-eb7c-4ca7-82ff-9452d51648b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"
Oct 30 20:02:41.946567 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.946545    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33874386-eb7c-4ca7-82ff-9452d51648b4-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55\" (UID: \"33874386-eb7c-4ca7-82ff-9452d51648b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"
Oct 30 20:02:41.985845 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:41.985793    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfgm8\" (UniqueName: \"kubernetes.io/projected/33874386-eb7c-4ca7-82ff-9452d51648b4-kube-api-access-gfgm8\") pod \"maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55\" (UID: \"33874386-eb7c-4ca7-82ff-9452d51648b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"
Oct 30 20:02:42.039987 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.039963    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"
Oct 30 20:02:42.122155 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.122125    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:42.253038 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.252966    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv"]
Oct 30 20:02:42.257592 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.257575    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv"
Oct 30 20:02:42.280203 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.280181    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-31a039-dockercfg-jscgw\""
Oct 30 20:02:42.280329 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.280203    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-31a039\""
Oct 30 20:02:42.282096 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.282071    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv"]
Oct 30 20:02:42.291605 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.291584    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:42.347220 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.347188    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96392fb-a2c5-4ec1-b5cb-a738529cc80f-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv\" (UID: \"c96392fb-a2c5-4ec1-b5cb-a738529cc80f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv"
Oct 30 20:02:42.347444 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.347234    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lzsd\" (UniqueName: \"kubernetes.io/projected/c96392fb-a2c5-4ec1-b5cb-a738529cc80f-kube-api-access-6lzsd\") pod \"payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv\" (UID: \"c96392fb-a2c5-4ec1-b5cb-a738529cc80f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv"
Oct 30 20:02:42.364304 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.364278    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c"]
Oct 30 20:02:42.365386 ip-10-0-140-248 kubenswrapper[4712]: W1030 20:02:42.365344    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33874386_eb7c_4ca7_82ff_9452d51648b4.slice/crio-f2d790163f79b696287fbf3930299d0dcb2c3fc7924366f14dcedf370524abaa WatchSource:0}: Error finding container f2d790163f79b696287fbf3930299d0dcb2c3fc7924366f14dcedf370524abaa: Status 404 returned error can't find the container with id f2d790163f79b696287fbf3930299d0dcb2c3fc7924366f14dcedf370524abaa
Oct 30 20:02:42.368274 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.368251    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"]
Oct 30 20:02:42.368394 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.368338    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c"
Oct 30 20:02:42.395581 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.395555    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c"]
Oct 30 20:02:42.447893 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.447868    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58997\" (UniqueName: \"kubernetes.io/projected/434349a4-e929-4c23-9ce4-4c418e94e466-kube-api-access-58997\") pod \"payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c\" (UID: \"434349a4-e929-4c23-9ce4-4c418e94e466\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c"
Oct 30 20:02:42.448002 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.447936    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96392fb-a2c5-4ec1-b5cb-a738529cc80f-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv\" (UID: \"c96392fb-a2c5-4ec1-b5cb-a738529cc80f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv"
Oct 30 20:02:42.448050 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.448003    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lzsd\" (UniqueName: \"kubernetes.io/projected/c96392fb-a2c5-4ec1-b5cb-a738529cc80f-kube-api-access-6lzsd\") pod \"payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv\" (UID: \"c96392fb-a2c5-4ec1-b5cb-a738529cc80f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv"
Oct 30 20:02:42.448050 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.448037    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434349a4-e929-4c23-9ce4-4c418e94e466-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c\" (UID: \"434349a4-e929-4c23-9ce4-4c418e94e466\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c"
Oct 30 20:02:42.448635 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.448613    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96392fb-a2c5-4ec1-b5cb-a738529cc80f-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv\" (UID: \"c96392fb-a2c5-4ec1-b5cb-a738529cc80f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv"
Oct 30 20:02:42.467313 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.467289    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lzsd\" (UniqueName: \"kubernetes.io/projected/c96392fb-a2c5-4ec1-b5cb-a738529cc80f-kube-api-access-6lzsd\") pod \"payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv\" (UID: \"c96392fb-a2c5-4ec1-b5cb-a738529cc80f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv"
Oct 30 20:02:42.548999 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.548948    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58997\" (UniqueName: \"kubernetes.io/projected/434349a4-e929-4c23-9ce4-4c418e94e466-kube-api-access-58997\") pod \"payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c\" (UID: \"434349a4-e929-4c23-9ce4-4c418e94e466\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c"
Oct 30 20:02:42.549088 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.549016    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434349a4-e929-4c23-9ce4-4c418e94e466-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c\" (UID: \"434349a4-e929-4c23-9ce4-4c418e94e466\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c"
Oct 30 20:02:42.549657 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.549640    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434349a4-e929-4c23-9ce4-4c418e94e466-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c\" (UID: \"434349a4-e929-4c23-9ce4-4c418e94e466\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c"
Oct 30 20:02:42.568176 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.568155    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58997\" (UniqueName: \"kubernetes.io/projected/434349a4-e929-4c23-9ce4-4c418e94e466-kube-api-access-58997\") pod \"payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c\" (UID: \"434349a4-e929-4c23-9ce4-4c418e94e466\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c"
Oct 30 20:02:42.570963 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.570946    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv"
Oct 30 20:02:42.681667 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.681639    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c"
Oct 30 20:02:42.850892 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.850818    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b26d2c-afba-44d7-8323-555420014ee1" path="/var/lib/kubelet/pods/a9b26d2c-afba-44d7-8323-555420014ee1/volumes"
Oct 30 20:02:42.854390 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.854346    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55" event={"ID":"33874386-eb7c-4ca7-82ff-9452d51648b4","Type":"ContainerStarted","Data":"52fe038f683e46a495d282f3b64d60cee032a313222ff3e7b10e224ada8ab75f"}
Oct 30 20:02:42.854537 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.854404    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55" event={"ID":"33874386-eb7c-4ca7-82ff-9452d51648b4","Type":"ContainerStarted","Data":"f2d790163f79b696287fbf3930299d0dcb2c3fc7924366f14dcedf370524abaa"}
Oct 30 20:02:42.854537 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.854521    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"
Oct 30 20:02:42.864455 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.864432    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv"]
Oct 30 20:02:42.865184 ip-10-0-140-248 kubenswrapper[4712]: W1030 20:02:42.865167    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96392fb_a2c5_4ec1_b5cb_a738529cc80f.slice/crio-30fd0e12aa46b789884ad75097457994446631cf58bb928db2018518fe60bf2a WatchSource:0}: Error finding container 30fd0e12aa46b789884ad75097457994446631cf58bb928db2018518fe60bf2a: Status 404 returned error can't find the container with id 30fd0e12aa46b789884ad75097457994446631cf58bb928db2018518fe60bf2a
Oct 30 20:02:42.895757 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.895711    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55" podStartSLOduration=1.555167569 podStartE2EDuration="1.895696096s" totalImagesPullingTime="340.528527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:42.893517627 +0000 UTC m=+2516.192779757" watchObservedRunningTime="2026-10-30 20:02:42.895696096 +0000 UTC m=+2516.194958224"
Oct 30 20:02:42.983934 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:42.983909    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c"]
Oct 30 20:02:42.984637 ip-10-0-140-248 kubenswrapper[4712]: W1030 20:02:42.984614    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434349a4_e929_4c23_9ce4_4c418e94e466.slice/crio-52553ece8d7591e3feaba74e66d70ff4bf779b04b4df726056ad24dcb8b853f0 WatchSource:0}: Error finding container 52553ece8d7591e3feaba74e66d70ff4bf779b04b4df726056ad24dcb8b853f0: Status 404 returned error can't find the container with id 52553ece8d7591e3feaba74e66d70ff4bf779b04b4df726056ad24dcb8b853f0
Oct 30 20:02:43.604463 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:43.604431    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:43.861086 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:43.861004    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv" event={"ID":"c96392fb-a2c5-4ec1-b5cb-a738529cc80f","Type":"ContainerStarted","Data":"da4a528e258e64f46bb4e923852f454d835af306e9e4a558eff457d93e9a789a"}
Oct 30 20:02:43.861086 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:43.861066    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv"
Oct 30 20:02:43.861555 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:43.861093    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv" event={"ID":"c96392fb-a2c5-4ec1-b5cb-a738529cc80f","Type":"ContainerStarted","Data":"30fd0e12aa46b789884ad75097457994446631cf58bb928db2018518fe60bf2a"}
Oct 30 20:02:43.862753 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:43.862730    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c" event={"ID":"434349a4-e929-4c23-9ce4-4c418e94e466","Type":"ContainerStarted","Data":"69863324f0648bd76fe3d453d4402250661b182fdec854303a89db959a2f6fa6"}
Oct 30 20:02:43.862860 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:43.862777    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c"
Oct 30 20:02:43.862860 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:43.862797    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c" event={"ID":"434349a4-e929-4c23-9ce4-4c418e94e466","Type":"ContainerStarted","Data":"52553ece8d7591e3feaba74e66d70ff4bf779b04b4df726056ad24dcb8b853f0"}
Oct 30 20:02:43.902225 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:43.902180    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv" podStartSLOduration=1.902168172 podStartE2EDuration="1.902168172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:43.900529924 +0000 UTC m=+2517.199792053" watchObservedRunningTime="2026-10-30 20:02:43.902168172 +0000 UTC m=+2517.201430301"
Oct 30 20:02:43.942539 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:43.942501    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c" podStartSLOduration=1.942488845 podStartE2EDuration="1.942488845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:43.941312046 +0000 UTC m=+2517.240574175" watchObservedRunningTime="2026-10-30 20:02:43.942488845 +0000 UTC m=+2517.241750974"
Oct 30 20:02:45.194496 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:45.194456    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:46.697353 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:46.697316    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:48.495657 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:48.495534    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:48.868919 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:48.868896    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"
Oct 30 20:02:50.037387 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:50.029389    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:51.619398 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:51.612484    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:53.166714 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:53.166673    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:54.704949 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:54.704912    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:54.869794 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:54.869768    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c"
Oct 30 20:02:54.869990 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:54.869836    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv"
Oct 30 20:02:55.299833 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:55.298897    4712 patch_prober.go:29] interesting pod/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": read tcp 10.133.0.2:53694->10.133.0.117:15021: read: connection reset by peer" start-of-body=
Oct 30 20:02:55.299833 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:55.298962    4712 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4" podUID="f3d7df99-9688-4e01-b42c-30bc86b89295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": read tcp 10.133.0.2:53694->10.133.0.117:15021: read: connection reset by peer"
Oct 30 20:02:55.335553 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:55.335528    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:55.916923 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:55.916892    4712 generic.go:356] "Generic (PLEG): container finished" podID="f3d7df99-9688-4e01-b42c-30bc86b89295" containerID="7f1ae2c2220bb304a6fbbd7589b656d4cb85a21443c91ca2056c8f3069a9fed7" exitCode=0
Oct 30 20:02:55.917312 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:55.916957    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4" event={"ID":"f3d7df99-9688-4e01-b42c-30bc86b89295","Type":"ContainerDied","Data":"7f1ae2c2220bb304a6fbbd7589b656d4cb85a21443c91ca2056c8f3069a9fed7"}
Oct 30 20:02:55.917312 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:55.917273    4712 scope.go:122] "RemoveContainer" containerID="7f1ae2c2220bb304a6fbbd7589b656d4cb85a21443c91ca2056c8f3069a9fed7"
Oct 30 20:02:55.918405 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:55.918358    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:02:55.918513 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:55.918467    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:02:55.918580 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:55.918520    4712 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 20:02:56.284901 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:56.284814    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:56.934251 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:56.934216    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4" event={"ID":"f3d7df99-9688-4e01-b42c-30bc86b89295","Type":"ContainerStarted","Data":"424ad511dc279bacf2aa84b5fdd2bea6bd273e4666b99a05a5a06fd9d7671c4f"}
Oct 30 20:02:57.835667 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:57.835633    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:02:57.835880 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:57.835824    4712 patch_prober.go:29] interesting pod/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:02:57.835880 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:57.835866    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4" podUID="f3d7df99-9688-4e01-b42c-30bc86b89295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 20:02:57.841771 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:57.841746    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:58.832790 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:58.832756    4712 patch_prober.go:29] interesting pod/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:02:58.833246 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:58.832802    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4" podUID="f3d7df99-9688-4e01-b42c-30bc86b89295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 20:02:59.494536 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:59.494506    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:02:59.832943 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:59.832858    4712 patch_prober.go:29] interesting pod/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:02:59.832943 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:02:59.832911    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4" podUID="f3d7df99-9688-4e01-b42c-30bc86b89295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 20:03:00.832962 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:00.832924    4712 patch_prober.go:29] interesting pod/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:03:00.833418 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:00.832978    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4" podUID="f3d7df99-9688-4e01-b42c-30bc86b89295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 20:03:01.088777 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:01.088685    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:01.832603 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:01.832562    4712 patch_prober.go:29] interesting pod/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 30 20:03:01.832786 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:01.832618    4712 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4" podUID="f3d7df99-9688-4e01-b42c-30bc86b89295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 30 20:03:02.494861 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:02.494831    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:02.837099 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:02.837027    4712 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:03:02.837293 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:02.837275    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:03:02.838020 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:02.837999    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:03:04.087333 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:04.087299    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:05.562669 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:05.562637    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:06.372822 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:03:06.372774    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 20:03:07.081096 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:07.081061    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:08.608761 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:08.608731    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:10.161331 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:10.161302    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:11.685610 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:11.685552    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:13.180252 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:13.180216    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:14.921898 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:14.921867    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:16.395937 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:16.395897    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:17.993780 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:17.993737    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:19.584021 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:19.583986    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:21.128033 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:21.127998    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:22.589079 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:22.589051    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:24.106773 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:24.106739    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:25.693817 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:25.693781    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:26.927031 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:26.926993    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:27.126586 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:27.126558    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:27.287453 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:27.287336    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:28.676696 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:28.676641    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:30.192644 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:30.192578    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:31.776547 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:31.776502    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:33.379785 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:33.379751    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:34.871609 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:34.871578    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:35.174697 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:35.174618    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:36.373669 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:03:36.373635    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 20:03:36.495202 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:36.495174    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:36.919454 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:36.919423    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:37.066949 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:37.066918    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:37.968205 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:37.968172    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:39.605236 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:39.605204    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:41.211990 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:41.211955    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:42.689779 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:42.689739    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:44.287729 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:44.287699    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:45.863530 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:45.863491    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:47.354021 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:47.353988    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:48.784605 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:48.784573    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:50.512435 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:50.512403    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:51.906068 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:51.906036    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:53.496302 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:53.496260    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:55.178512 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:55.178481    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:56.783527 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:56.783498    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:58.301143 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:58.301104    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:03:59.776064 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:03:59.776030    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:01.372575 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:01.372533    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:02.970896 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:02.970861    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:04.627599 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:04.627563    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:06.081437 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:06.081386    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:06.372750 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:04:06.372716    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 20:04:07.681681 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:07.681633    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:09.252616 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:09.252585    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:10.676872 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:10.676838    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:12.129786 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:12.129757    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:13.609419 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:13.609342    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:15.020559 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:15.020517    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:15.292646 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:15.292567    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:16.949560 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:16.949526    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:18.485104 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:18.485073    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:19.979717 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:19.979683    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:21.578855 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:21.578825    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:23.123596 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:23.123563    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:24.603707 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:24.603656    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:25.821420 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:25.821385    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c"]
Oct 30 20:04:25.854749 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:25.854715    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv"]
Oct 30 20:04:26.001750 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:26.001719    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:26.150731 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:26.150700    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:26.310462 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:26.310429    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:30.822483 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:30.822429    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c" podUID="434349a4-e929-4c23-9ce4-4c418e94e466" containerName="payload-processing" containerID="cri-o://69863324f0648bd76fe3d453d4402250661b182fdec854303a89db959a2f6fa6" gracePeriod=25
Oct 30 20:04:30.855445 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:30.855394    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv" podUID="c96392fb-a2c5-4ec1-b5cb-a738529cc80f" containerName="payload-pre-processing" containerID="cri-o://da4a528e258e64f46bb4e923852f454d835af306e9e4a558eff457d93e9a789a" gracePeriod=25
Oct 30 20:04:31.092001 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.091978    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c"
Oct 30 20:04:31.118915 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.118895    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv"
Oct 30 20:04:31.236890 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.236859    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c96392fb-a2c5-4ec1-b5cb-a738529cc80f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96392fb-a2c5-4ec1-b5cb-a738529cc80f-plugins-config-volume\") pod \"c96392fb-a2c5-4ec1-b5cb-a738529cc80f\" (UID: \"c96392fb-a2c5-4ec1-b5cb-a738529cc80f\") "
Oct 30 20:04:31.237042 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.236961    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/434349a4-e929-4c23-9ce4-4c418e94e466-kube-api-access-58997\" (UniqueName: \"kubernetes.io/projected/434349a4-e929-4c23-9ce4-4c418e94e466-kube-api-access-58997\") pod \"434349a4-e929-4c23-9ce4-4c418e94e466\" (UID: \"434349a4-e929-4c23-9ce4-4c418e94e466\") "
Oct 30 20:04:31.237091 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.237059    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/434349a4-e929-4c23-9ce4-4c418e94e466-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434349a4-e929-4c23-9ce4-4c418e94e466-plugins-config-volume\") pod \"434349a4-e929-4c23-9ce4-4c418e94e466\" (UID: \"434349a4-e929-4c23-9ce4-4c418e94e466\") "
Oct 30 20:04:31.237146 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.237129    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c96392fb-a2c5-4ec1-b5cb-a738529cc80f-kube-api-access-6lzsd\" (UniqueName: \"kubernetes.io/projected/c96392fb-a2c5-4ec1-b5cb-a738529cc80f-kube-api-access-6lzsd\") pod \"c96392fb-a2c5-4ec1-b5cb-a738529cc80f\" (UID: \"c96392fb-a2c5-4ec1-b5cb-a738529cc80f\") "
Oct 30 20:04:31.237441 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.237415    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c96392fb-a2c5-4ec1-b5cb-a738529cc80f-plugins-config-volume" pod "c96392fb-a2c5-4ec1-b5cb-a738529cc80f" (UID: "c96392fb-a2c5-4ec1-b5cb-a738529cc80f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:31.237542 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.237463    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434349a4-e929-4c23-9ce4-4c418e94e466-plugins-config-volume" pod "434349a4-e929-4c23-9ce4-4c418e94e466" (UID: "434349a4-e929-4c23-9ce4-4c418e94e466"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:31.239214 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.239185    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c96392fb-a2c5-4ec1-b5cb-a738529cc80f-kube-api-access-6lzsd" pod "c96392fb-a2c5-4ec1-b5cb-a738529cc80f" (UID: "c96392fb-a2c5-4ec1-b5cb-a738529cc80f"). InnerVolumeSpecName "kube-api-access-6lzsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:31.239313 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.239192    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434349a4-e929-4c23-9ce4-4c418e94e466-kube-api-access-58997" pod "434349a4-e929-4c23-9ce4-4c418e94e466" (UID: "434349a4-e929-4c23-9ce4-4c418e94e466"). InnerVolumeSpecName "kube-api-access-58997". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:31.338550 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.338491    4712 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434349a4-e929-4c23-9ce4-4c418e94e466-plugins-config-volume\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:04:31.338550 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.338516    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lzsd\" (UniqueName: \"kubernetes.io/projected/c96392fb-a2c5-4ec1-b5cb-a738529cc80f-kube-api-access-6lzsd\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:04:31.338550 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.338532    4712 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96392fb-a2c5-4ec1-b5cb-a738529cc80f-plugins-config-volume\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:04:31.338705 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.338546    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58997\" (UniqueName: \"kubernetes.io/projected/434349a4-e929-4c23-9ce4-4c418e94e466-kube-api-access-58997\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:04:31.352452 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.352421    4712 generic.go:356] "Generic (PLEG): container finished" podID="434349a4-e929-4c23-9ce4-4c418e94e466" containerID="69863324f0648bd76fe3d453d4402250661b182fdec854303a89db959a2f6fa6" exitCode=0
Oct 30 20:04:31.352549 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.352472    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c"
Oct 30 20:04:31.352549 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.352503    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c" event={"ID":"434349a4-e929-4c23-9ce4-4c418e94e466","Type":"ContainerDied","Data":"69863324f0648bd76fe3d453d4402250661b182fdec854303a89db959a2f6fa6"}
Oct 30 20:04:31.352646 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.352554    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c" event={"ID":"434349a4-e929-4c23-9ce4-4c418e94e466","Type":"ContainerDied","Data":"52553ece8d7591e3feaba74e66d70ff4bf779b04b4df726056ad24dcb8b853f0"}
Oct 30 20:04:31.352646 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.352580    4712 scope.go:122] "RemoveContainer" containerID="69863324f0648bd76fe3d453d4402250661b182fdec854303a89db959a2f6fa6"
Oct 30 20:04:31.353953 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.353933    4712 generic.go:356] "Generic (PLEG): container finished" podID="c96392fb-a2c5-4ec1-b5cb-a738529cc80f" containerID="da4a528e258e64f46bb4e923852f454d835af306e9e4a558eff457d93e9a789a" exitCode=0
Oct 30 20:04:31.354050 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.353991    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv"
Oct 30 20:04:31.354050 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.353997    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv" event={"ID":"c96392fb-a2c5-4ec1-b5cb-a738529cc80f","Type":"ContainerDied","Data":"da4a528e258e64f46bb4e923852f454d835af306e9e4a558eff457d93e9a789a"}
Oct 30 20:04:31.354050 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.354037    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv" event={"ID":"c96392fb-a2c5-4ec1-b5cb-a738529cc80f","Type":"ContainerDied","Data":"30fd0e12aa46b789884ad75097457994446631cf58bb928db2018518fe60bf2a"}
Oct 30 20:04:31.361013 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.360996    4712 scope.go:122] "RemoveContainer" containerID="69863324f0648bd76fe3d453d4402250661b182fdec854303a89db959a2f6fa6"
Oct 30 20:04:31.361286 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:04:31.361268    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69863324f0648bd76fe3d453d4402250661b182fdec854303a89db959a2f6fa6\": container with ID starting with 69863324f0648bd76fe3d453d4402250661b182fdec854303a89db959a2f6fa6 not found: ID does not exist" containerID="69863324f0648bd76fe3d453d4402250661b182fdec854303a89db959a2f6fa6"
Oct 30 20:04:31.361350 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.361294    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69863324f0648bd76fe3d453d4402250661b182fdec854303a89db959a2f6fa6"} err="failed to get container status \"69863324f0648bd76fe3d453d4402250661b182fdec854303a89db959a2f6fa6\": rpc error: code = NotFound desc = could not find container \"69863324f0648bd76fe3d453d4402250661b182fdec854303a89db959a2f6fa6\": container with ID starting with 69863324f0648bd76fe3d453d4402250661b182fdec854303a89db959a2f6fa6 not found: ID does not exist"
Oct 30 20:04:31.361350 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.361310    4712 scope.go:122] "RemoveContainer" containerID="da4a528e258e64f46bb4e923852f454d835af306e9e4a558eff457d93e9a789a"
Oct 30 20:04:31.371785 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.371768    4712 scope.go:122] "RemoveContainer" containerID="da4a528e258e64f46bb4e923852f454d835af306e9e4a558eff457d93e9a789a"
Oct 30 20:04:31.372071 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:04:31.372050    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da4a528e258e64f46bb4e923852f454d835af306e9e4a558eff457d93e9a789a\": container with ID starting with da4a528e258e64f46bb4e923852f454d835af306e9e4a558eff457d93e9a789a not found: ID does not exist" containerID="da4a528e258e64f46bb4e923852f454d835af306e9e4a558eff457d93e9a789a"
Oct 30 20:04:31.372206 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.372074    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da4a528e258e64f46bb4e923852f454d835af306e9e4a558eff457d93e9a789a"} err="failed to get container status \"da4a528e258e64f46bb4e923852f454d835af306e9e4a558eff457d93e9a789a\": rpc error: code = NotFound desc = could not find container \"da4a528e258e64f46bb4e923852f454d835af306e9e4a558eff457d93e9a789a\": container with ID starting with da4a528e258e64f46bb4e923852f454d835af306e9e4a558eff457d93e9a789a not found: ID does not exist"
Oct 30 20:04:31.406643 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.406618    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv"]
Oct 30 20:04:31.411074 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.411051    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-31a039-99789fc5f-p8bgv"]
Oct 30 20:04:31.444684 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.444662    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c"]
Oct 30 20:04:31.448867 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:31.448844    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-31a039-6945c4f4b6-74d7c"]
Oct 30 20:04:32.399498 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.399466    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"]
Oct 30 20:04:32.400008 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.399704    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55" podUID="33874386-eb7c-4ca7-82ff-9452d51648b4" containerName="maas-api" containerID="cri-o://52fe038f683e46a495d282f3b64d60cee032a313222ff3e7b10e224ada8ab75f" gracePeriod=30
Oct 30 20:04:32.534817 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.534788    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:32.645635 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.645603    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"
Oct 30 20:04:32.650013 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.649944    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33874386-eb7c-4ca7-82ff-9452d51648b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33874386-eb7c-4ca7-82ff-9452d51648b4-metrics-certs\") pod \"33874386-eb7c-4ca7-82ff-9452d51648b4\" (UID: \"33874386-eb7c-4ca7-82ff-9452d51648b4\") "
Oct 30 20:04:32.650013 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.649998    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33874386-eb7c-4ca7-82ff-9452d51648b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33874386-eb7c-4ca7-82ff-9452d51648b4-maas-api-tls\") pod \"33874386-eb7c-4ca7-82ff-9452d51648b4\" (UID: \"33874386-eb7c-4ca7-82ff-9452d51648b4\") "
Oct 30 20:04:32.650256 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.650076    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33874386-eb7c-4ca7-82ff-9452d51648b4-kube-api-access-gfgm8\" (UniqueName: \"kubernetes.io/projected/33874386-eb7c-4ca7-82ff-9452d51648b4-kube-api-access-gfgm8\") pod \"33874386-eb7c-4ca7-82ff-9452d51648b4\" (UID: \"33874386-eb7c-4ca7-82ff-9452d51648b4\") "
Oct 30 20:04:32.650357 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.650251    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33874386-eb7c-4ca7-82ff-9452d51648b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33874386-eb7c-4ca7-82ff-9452d51648b4-openshift-service-ca\") pod \"33874386-eb7c-4ca7-82ff-9452d51648b4\" (UID: \"33874386-eb7c-4ca7-82ff-9452d51648b4\") "
Oct 30 20:04:32.651116 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.651067    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33874386-eb7c-4ca7-82ff-9452d51648b4-openshift-service-ca" pod "33874386-eb7c-4ca7-82ff-9452d51648b4" (UID: "33874386-eb7c-4ca7-82ff-9452d51648b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:32.653134 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.653098    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33874386-eb7c-4ca7-82ff-9452d51648b4-metrics-certs" pod "33874386-eb7c-4ca7-82ff-9452d51648b4" (UID: "33874386-eb7c-4ca7-82ff-9452d51648b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:32.653476 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.653447    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33874386-eb7c-4ca7-82ff-9452d51648b4-maas-api-tls" pod "33874386-eb7c-4ca7-82ff-9452d51648b4" (UID: "33874386-eb7c-4ca7-82ff-9452d51648b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:32.653715 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.653681    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33874386-eb7c-4ca7-82ff-9452d51648b4-kube-api-access-gfgm8" pod "33874386-eb7c-4ca7-82ff-9452d51648b4" (UID: "33874386-eb7c-4ca7-82ff-9452d51648b4"). InnerVolumeSpecName "kube-api-access-gfgm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:32.752810 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.752781    4712 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33874386-eb7c-4ca7-82ff-9452d51648b4-maas-api-tls\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:04:32.752810 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.752811    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfgm8\" (UniqueName: \"kubernetes.io/projected/33874386-eb7c-4ca7-82ff-9452d51648b4-kube-api-access-gfgm8\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:04:32.753003 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.752826    4712 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33874386-eb7c-4ca7-82ff-9452d51648b4-openshift-service-ca\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:04:32.753003 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.752839    4712 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33874386-eb7c-4ca7-82ff-9452d51648b4-metrics-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:04:32.854802 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.854768    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="434349a4-e929-4c23-9ce4-4c418e94e466" path="/var/lib/kubelet/pods/434349a4-e929-4c23-9ce4-4c418e94e466/volumes"
Oct 30 20:04:32.855179 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.855158    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c96392fb-a2c5-4ec1-b5cb-a738529cc80f" path="/var/lib/kubelet/pods/c96392fb-a2c5-4ec1-b5cb-a738529cc80f/volumes"
Oct 30 20:04:32.995678 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:32.995590    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:33.161060 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:33.161020    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:33.365201 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:33.365120    4712 generic.go:356] "Generic (PLEG): container finished" podID="33874386-eb7c-4ca7-82ff-9452d51648b4" containerID="52fe038f683e46a495d282f3b64d60cee032a313222ff3e7b10e224ada8ab75f" exitCode=0
Oct 30 20:04:33.365201 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:33.365163    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"
Oct 30 20:04:33.365433 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:33.365208    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55" event={"ID":"33874386-eb7c-4ca7-82ff-9452d51648b4","Type":"ContainerDied","Data":"52fe038f683e46a495d282f3b64d60cee032a313222ff3e7b10e224ada8ab75f"}
Oct 30 20:04:33.365433 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:33.365255    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55" event={"ID":"33874386-eb7c-4ca7-82ff-9452d51648b4","Type":"ContainerDied","Data":"f2d790163f79b696287fbf3930299d0dcb2c3fc7924366f14dcedf370524abaa"}
Oct 30 20:04:33.365433 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:33.365277    4712 scope.go:122] "RemoveContainer" containerID="52fe038f683e46a495d282f3b64d60cee032a313222ff3e7b10e224ada8ab75f"
Oct 30 20:04:33.373883 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:33.373852    4712 scope.go:122] "RemoveContainer" containerID="52fe038f683e46a495d282f3b64d60cee032a313222ff3e7b10e224ada8ab75f"
Oct 30 20:04:33.374132 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:04:33.374114    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52fe038f683e46a495d282f3b64d60cee032a313222ff3e7b10e224ada8ab75f\": container with ID starting with 52fe038f683e46a495d282f3b64d60cee032a313222ff3e7b10e224ada8ab75f not found: ID does not exist" containerID="52fe038f683e46a495d282f3b64d60cee032a313222ff3e7b10e224ada8ab75f"
Oct 30 20:04:33.374177 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:33.374145    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52fe038f683e46a495d282f3b64d60cee032a313222ff3e7b10e224ada8ab75f"} err="failed to get container status \"52fe038f683e46a495d282f3b64d60cee032a313222ff3e7b10e224ada8ab75f\": rpc error: code = NotFound desc = could not find container \"52fe038f683e46a495d282f3b64d60cee032a313222ff3e7b10e224ada8ab75f\": container with ID starting with 52fe038f683e46a495d282f3b64d60cee032a313222ff3e7b10e224ada8ab75f not found: ID does not exist"
Oct 30 20:04:33.414936 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:33.414911    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"]
Oct 30 20:04:33.422397 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:33.422355    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-31a039-7789f6c47c-lxf55"]
Oct 30 20:04:33.652007 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:33.651976    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:34.850869 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:34.850834    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33874386-eb7c-4ca7-82ff-9452d51648b4" path="/var/lib/kubelet/pods/33874386-eb7c-4ca7-82ff-9452d51648b4/volumes"
Oct 30 20:04:35.287301 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:35.287265    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:36.372779 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:04:36.372745    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 20:04:39.577987 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:39.577956    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:39.728282 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:39.728248    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:39.864777 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:39.864682    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"]
Oct 30 20:04:39.865080 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:39.865045    4712 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4" podUID="f3d7df99-9688-4e01-b42c-30bc86b89295" containerName="istio-proxy" containerID="cri-o://424ad511dc279bacf2aa84b5fdd2bea6bd273e4666b99a05a5a06fd9d7671c4f" gracePeriod=30
Oct 30 20:04:39.955945 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:39.955919    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:04:45.320384 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.318071    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:04:45.402105 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.402077    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-credential-socket\") pod \"f3d7df99-9688-4e01-b42c-30bc86b89295\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") "
Oct 30 20:04:45.402290 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.402113    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-token\" (UniqueName: \"kubernetes.io/projected/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-token\") pod \"f3d7df99-9688-4e01-b42c-30bc86b89295\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") "
Oct 30 20:04:45.402290 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.402149    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-workload-certs\") pod \"f3d7df99-9688-4e01-b42c-30bc86b89295\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") "
Oct 30 20:04:45.402290 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.402267    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-workload-socket\") pod \"f3d7df99-9688-4e01-b42c-30bc86b89295\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") "
Oct 30 20:04:45.402536 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.402346    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-envoy\") pod \"f3d7df99-9688-4e01-b42c-30bc86b89295\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") "
Oct 30 20:04:45.402536 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.402420    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-data\") pod \"f3d7df99-9688-4e01-b42c-30bc86b89295\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") "
Oct 30 20:04:45.402536 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.402411    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-credential-socket" pod "f3d7df99-9688-4e01-b42c-30bc86b89295" (UID: "f3d7df99-9688-4e01-b42c-30bc86b89295"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:45.402536 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.402447    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-workload-certs" pod "f3d7df99-9688-4e01-b42c-30bc86b89295" (UID: "f3d7df99-9688-4e01-b42c-30bc86b89295"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:45.402536 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.402453    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3d7df99-9688-4e01-b42c-30bc86b89295-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d7df99-9688-4e01-b42c-30bc86b89295-istiod-ca-cert\") pod \"f3d7df99-9688-4e01-b42c-30bc86b89295\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") "
Oct 30 20:04:45.402536 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.402487    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3d7df99-9688-4e01-b42c-30bc86b89295-kube-api-access-7z4v5\" (UniqueName: \"kubernetes.io/projected/f3d7df99-9688-4e01-b42c-30bc86b89295-kube-api-access-7z4v5\") pod \"f3d7df99-9688-4e01-b42c-30bc86b89295\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") "
Oct 30 20:04:45.402767 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.402538    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-workload-socket" pod "f3d7df99-9688-4e01-b42c-30bc86b89295" (UID: "f3d7df99-9688-4e01-b42c-30bc86b89295"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:45.402767 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.402637    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-data" pod "f3d7df99-9688-4e01-b42c-30bc86b89295" (UID: "f3d7df99-9688-4e01-b42c-30bc86b89295"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:45.402924 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.402814    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3d7df99-9688-4e01-b42c-30bc86b89295-istiod-ca-cert" pod "f3d7df99-9688-4e01-b42c-30bc86b89295" (UID: "f3d7df99-9688-4e01-b42c-30bc86b89295"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:45.402924 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.402881    4712 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-data\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:04:45.402924 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.402902    4712 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-credential-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:04:45.402924 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.402916    4712 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-workload-certs\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:04:45.403136 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.402929    4712 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-workload-socket\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:04:45.420611 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.420472    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-envoy" pod "f3d7df99-9688-4e01-b42c-30bc86b89295" (UID: "f3d7df99-9688-4e01-b42c-30bc86b89295"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:45.420611 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.420565    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d7df99-9688-4e01-b42c-30bc86b89295-kube-api-access-7z4v5" pod "f3d7df99-9688-4e01-b42c-30bc86b89295" (UID: "f3d7df99-9688-4e01-b42c-30bc86b89295"). InnerVolumeSpecName "kube-api-access-7z4v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:45.422643 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.422471    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-token" pod "f3d7df99-9688-4e01-b42c-30bc86b89295" (UID: "f3d7df99-9688-4e01-b42c-30bc86b89295"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:45.464157 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.463293    4712 generic.go:356] "Generic (PLEG): container finished" podID="f3d7df99-9688-4e01-b42c-30bc86b89295" containerID="424ad511dc279bacf2aa84b5fdd2bea6bd273e4666b99a05a5a06fd9d7671c4f" exitCode=0
Oct 30 20:04:45.464157 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.463388    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4" event={"ID":"f3d7df99-9688-4e01-b42c-30bc86b89295","Type":"ContainerDied","Data":"424ad511dc279bacf2aa84b5fdd2bea6bd273e4666b99a05a5a06fd9d7671c4f"}
Oct 30 20:04:45.464157 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.463439    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4" event={"ID":"f3d7df99-9688-4e01-b42c-30bc86b89295","Type":"ContainerDied","Data":"1a6a48fd3fcd42c6ce6d29dfd3f7fb3d474c88311cc4c0ac21afabac15387218"}
Oct 30 20:04:45.464157 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.463471    4712 scope.go:122] "RemoveContainer" containerID="424ad511dc279bacf2aa84b5fdd2bea6bd273e4666b99a05a5a06fd9d7671c4f"
Oct 30 20:04:45.464157 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.463630    4712 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"
Oct 30 20:04:45.483885 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.483868    4712 scope.go:122] "RemoveContainer" containerID="7f1ae2c2220bb304a6fbbd7589b656d4cb85a21443c91ca2056c8f3069a9fed7"
Oct 30 20:04:45.496494 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.496418    4712 scope.go:122] "RemoveContainer" containerID="424ad511dc279bacf2aa84b5fdd2bea6bd273e4666b99a05a5a06fd9d7671c4f"
Oct 30 20:04:45.496749 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:04:45.496729    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"424ad511dc279bacf2aa84b5fdd2bea6bd273e4666b99a05a5a06fd9d7671c4f\": container with ID starting with 424ad511dc279bacf2aa84b5fdd2bea6bd273e4666b99a05a5a06fd9d7671c4f not found: ID does not exist" containerID="424ad511dc279bacf2aa84b5fdd2bea6bd273e4666b99a05a5a06fd9d7671c4f"
Oct 30 20:04:45.496821 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.496759    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"424ad511dc279bacf2aa84b5fdd2bea6bd273e4666b99a05a5a06fd9d7671c4f"} err="failed to get container status \"424ad511dc279bacf2aa84b5fdd2bea6bd273e4666b99a05a5a06fd9d7671c4f\": rpc error: code = NotFound desc = could not find container \"424ad511dc279bacf2aa84b5fdd2bea6bd273e4666b99a05a5a06fd9d7671c4f\": container with ID starting with 424ad511dc279bacf2aa84b5fdd2bea6bd273e4666b99a05a5a06fd9d7671c4f not found: ID does not exist"
Oct 30 20:04:45.496821 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.496776    4712 scope.go:122] "RemoveContainer" containerID="7f1ae2c2220bb304a6fbbd7589b656d4cb85a21443c91ca2056c8f3069a9fed7"
Oct 30 20:04:45.497062 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:04:45.497043    4712 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f1ae2c2220bb304a6fbbd7589b656d4cb85a21443c91ca2056c8f3069a9fed7\": container with ID starting with 7f1ae2c2220bb304a6fbbd7589b656d4cb85a21443c91ca2056c8f3069a9fed7 not found: ID does not exist" containerID="7f1ae2c2220bb304a6fbbd7589b656d4cb85a21443c91ca2056c8f3069a9fed7"
Oct 30 20:04:45.497127 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.497073    4712 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f1ae2c2220bb304a6fbbd7589b656d4cb85a21443c91ca2056c8f3069a9fed7"} err="failed to get container status \"7f1ae2c2220bb304a6fbbd7589b656d4cb85a21443c91ca2056c8f3069a9fed7\": rpc error: code = NotFound desc = could not find container \"7f1ae2c2220bb304a6fbbd7589b656d4cb85a21443c91ca2056c8f3069a9fed7\": container with ID starting with 7f1ae2c2220bb304a6fbbd7589b656d4cb85a21443c91ca2056c8f3069a9fed7 not found: ID does not exist"
Oct 30 20:04:45.503498 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.503478    4712 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-podinfo\") pod \"f3d7df99-9688-4e01-b42c-30bc86b89295\" (UID: \"f3d7df99-9688-4e01-b42c-30bc86b89295\") "
Oct 30 20:04:45.503741 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.503726    4712 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-token\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:04:45.503808 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.503746    4712 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-envoy\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:04:45.503808 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.503758    4712 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d7df99-9688-4e01-b42c-30bc86b89295-istiod-ca-cert\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:04:45.503808 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.503772    4712 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z4v5\" (UniqueName: \"kubernetes.io/projected/f3d7df99-9688-4e01-b42c-30bc86b89295-kube-api-access-7z4v5\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:04:45.536516 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.536485    4712 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-podinfo" pod "f3d7df99-9688-4e01-b42c-30bc86b89295" (UID: "f3d7df99-9688-4e01-b42c-30bc86b89295"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:04:45.605287 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.605212    4712 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d7df99-9688-4e01-b42c-30bc86b89295-istio-podinfo\") on node \"ip-10-0-140-248.ec2.internal\" DevicePath \"\""
Oct 30 20:04:45.808127 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.808096    4712 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"]
Oct 30 20:04:45.812141 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:45.812119    4712 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-31a039-openshift-default-b6995cf85-wq8l4"]
Oct 30 20:04:46.850286 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:04:46.850247    4712 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3d7df99-9688-4e01-b42c-30bc86b89295" path="/var/lib/kubelet/pods/f3d7df99-9688-4e01-b42c-30bc86b89295/volumes"
Oct 30 20:05:06.372411 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:05:06.372314    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 20:05:35.417771 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:05:35.417735    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:05:36.372229 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:05:36.372189    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 20:05:47.176427 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:05:47.176399    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 20:05:47.179385 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:05:47.179350    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6rkvc_8c299f86-7f6e-4666-ab9c-542537e6ff8e/ovn-acl-logging/0.log"
Oct 30 20:05:47.545047 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:05:47.544972    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-744cf976bc-l8stx_916569a8-b32b-410a-b619-cd522570fbd9/manager/0.log"
Oct 30 20:05:50.056728 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:05:50.056693    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-9dcd87b65-r5rpr_84b4d803-9fa8-438c-9c08-094eb6405987/maas-api/0.log"
Oct 30 20:05:53.669846 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:05:53.669817    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-fgcgf_f055912e-ab5f-4098-ba65-bc664b7aee30/kuadrant-console-plugin/0.log"
Oct 30 20:05:55.745666 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:05:55.745629    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58487bf9b4-lck4n_cd1c6ba9-e503-4755-89c2-1700e80c4e12/kube-auth-proxy/0.log"
Oct 30 20:05:55.960703 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:05:55.960674    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-28vt6_8cbf06ec-e967-4cc4-9d06-6629e3be62da/istio-proxy/0.log"
Oct 30 20:05:56.754905 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:05:56.754865    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-cdt2t_f1c940f3-a6ba-45e5-bb26-2997810c97b6/istio-proxy/0.log"
Oct 30 20:05:57.361948 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:05:57.361917    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-rflrd_75fe15f5-086d-455e-af75-a99955780e72/istio-proxy/0.log"
Oct 30 20:05:58.120410 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:05:58.120380    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-7p7sg_17538073-ba9e-44e4-98b1-80a78223f37d/payload-pre-processing/0.log"
Oct 30 20:05:59.681986 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:05:59.681962    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-k2ndr_45dec9f8-1002-4ccc-b536-d43e3c3cfbb4/main/0.log"
Oct 30 20:06:00.101757 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:00.101674    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-xzskw_06773fac-ac83-4d67-902f-da073082dd63/main/0.log"
Oct 30 20:06:00.318116 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:00.318088    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n_78c9b32e-faf1-442c-ba28-30d14902258e/main/0.log"
Oct 30 20:06:00.509316 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:00.509289    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-2n6zs_de5e94b0-0bc5-4fe0-afce-708bbbfd073b/main/0.log"
Oct 30 20:06:00.718849 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:00.718818    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt_038379f4-4d41-4502-8142-df1e91332286/main/0.log"
Oct 30 20:06:00.927834 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:00.927803    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc_fd766cbc-5850-4181-95fa-a494ed7c8736/main/0.log"
Oct 30 20:06:06.372568 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:06:06.372531    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 20:06:11.183987 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:11.183958    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-744cf976bc-l8stx_916569a8-b32b-410a-b619-cd522570fbd9/manager/0.log"
Oct 30 20:06:13.481000 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:13.480972    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-9dcd87b65-r5rpr_84b4d803-9fa8-438c-9c08-094eb6405987/maas-api/0.log"
Oct 30 20:06:16.393494 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:16.393466    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-fgcgf_f055912e-ab5f-4098-ba65-bc664b7aee30/kuadrant-console-plugin/0.log"
Oct 30 20:06:18.199310 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:18.199284    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58487bf9b4-lck4n_cd1c6ba9-e503-4755-89c2-1700e80c4e12/kube-auth-proxy/0.log"
Oct 30 20:06:18.400538 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:18.400499    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-28vt6_8cbf06ec-e967-4cc4-9d06-6629e3be62da/istio-proxy/0.log"
Oct 30 20:06:19.180028 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:19.179995    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-cdt2t_f1c940f3-a6ba-45e5-bb26-2997810c97b6/istio-proxy/0.log"
Oct 30 20:06:19.791672 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:19.791639    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-rflrd_75fe15f5-086d-455e-af75-a99955780e72/istio-proxy/0.log"
Oct 30 20:06:20.569064 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:20.569026    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-7p7sg_17538073-ba9e-44e4-98b1-80a78223f37d/payload-pre-processing/0.log"
Oct 30 20:06:21.703167 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:21.703138    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-k2ndr_45dec9f8-1002-4ccc-b536-d43e3c3cfbb4/main/0.log"
Oct 30 20:06:22.106902 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:22.106824    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-xzskw_06773fac-ac83-4d67-902f-da073082dd63/main/0.log"
Oct 30 20:06:22.318190 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:22.318162    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b895g5n_78c9b32e-faf1-442c-ba28-30d14902258e/main/0.log"
Oct 30 20:06:22.515913 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:22.515885    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-2n6zs_de5e94b0-0bc5-4fe0-afce-708bbbfd073b/main/0.log"
Oct 30 20:06:22.707447 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:22.707420    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-h8prt_038379f4-4d41-4502-8142-df1e91332286/main/0.log"
Oct 30 20:06:22.894167 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:22.894142    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-xmngc_fd766cbc-5850-4181-95fa-a494ed7c8736/main/0.log"
Oct 30 20:06:35.523935 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:06:35.523851    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:06:36.372538 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:06:36.372502    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 20:07:03.014902 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:03.014868    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:07:05.270618 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:05.270584    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:07:06.372598 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:07:06.372560    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 20:07:19.849778 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:19.849746    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:07:20.005874 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:20.005845    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:07:35.624577 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:35.624546    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:07:36.371782 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:07:36.371746    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 20:07:43.150194 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.150150    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9zq2q/must-gather-hvnqd"]
Oct 30 20:07:43.150702 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.150524    4712 cpu_manager.go:414] "containerMap: removing container" podUID="f3d7df99-9688-4e01-b42c-30bc86b89295" containerName="istio-proxy"
Oct 30 20:07:43.150702 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.150536    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3d7df99-9688-4e01-b42c-30bc86b89295" containerName="istio-proxy"
Oct 30 20:07:43.150702 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.150549    4712 cpu_manager.go:414] "containerMap: removing container" podUID="434349a4-e929-4c23-9ce4-4c418e94e466" containerName="payload-processing"
Oct 30 20:07:43.150702 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.150554    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="434349a4-e929-4c23-9ce4-4c418e94e466" containerName="payload-processing"
Oct 30 20:07:43.150702 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.150566    4712 cpu_manager.go:414] "containerMap: removing container" podUID="33874386-eb7c-4ca7-82ff-9452d51648b4" containerName="maas-api"
Oct 30 20:07:43.150702 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.150572    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33874386-eb7c-4ca7-82ff-9452d51648b4" containerName="maas-api"
Oct 30 20:07:43.150702 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.150585    4712 cpu_manager.go:414] "containerMap: removing container" podUID="c96392fb-a2c5-4ec1-b5cb-a738529cc80f" containerName="payload-pre-processing"
Oct 30 20:07:43.150702 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.150589    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c96392fb-a2c5-4ec1-b5cb-a738529cc80f" containerName="payload-pre-processing"
Oct 30 20:07:43.150702 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.150667    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3d7df99-9688-4e01-b42c-30bc86b89295" containerName="istio-proxy"
Oct 30 20:07:43.150702 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.150675    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="434349a4-e929-4c23-9ce4-4c418e94e466" containerName="payload-processing"
Oct 30 20:07:43.150702 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.150682    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="33874386-eb7c-4ca7-82ff-9452d51648b4" containerName="maas-api"
Oct 30 20:07:43.150702 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.150690    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="c96392fb-a2c5-4ec1-b5cb-a738529cc80f" containerName="payload-pre-processing"
Oct 30 20:07:43.151076 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.150751    4712 cpu_manager.go:414] "containerMap: removing container" podUID="f3d7df99-9688-4e01-b42c-30bc86b89295" containerName="istio-proxy"
Oct 30 20:07:43.151076 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.150757    4712 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3d7df99-9688-4e01-b42c-30bc86b89295" containerName="istio-proxy"
Oct 30 20:07:43.151076 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.150824    4712 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3d7df99-9688-4e01-b42c-30bc86b89295" containerName="istio-proxy"
Oct 30 20:07:43.153856 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.153838    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zq2q/must-gather-hvnqd"
Oct 30 20:07:43.183401 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.183380    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9zq2q\"/\"openshift-service-ca.crt\""
Oct 30 20:07:43.183530 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.183399    4712 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9zq2q\"/\"default-dockercfg-rgcbq\""
Oct 30 20:07:43.183530 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.183412    4712 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9zq2q\"/\"kube-root-ca.crt\""
Oct 30 20:07:43.217464 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.217440    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zq2q/must-gather-hvnqd"]
Oct 30 20:07:43.243875 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.243844    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:07:43.294574 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.294548    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlglx\" (UniqueName: \"kubernetes.io/projected/3686e231-f63d-4889-9fd6-c137e1584028-kube-api-access-wlglx\") pod \"must-gather-hvnqd\" (UID: \"3686e231-f63d-4889-9fd6-c137e1584028\") " pod="openshift-must-gather-9zq2q/must-gather-hvnqd"
Oct 30 20:07:43.294688 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.294583    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3686e231-f63d-4889-9fd6-c137e1584028-must-gather-output\") pod \"must-gather-hvnqd\" (UID: \"3686e231-f63d-4889-9fd6-c137e1584028\") " pod="openshift-must-gather-9zq2q/must-gather-hvnqd"
Oct 30 20:07:43.395786 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.395758    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlglx\" (UniqueName: \"kubernetes.io/projected/3686e231-f63d-4889-9fd6-c137e1584028-kube-api-access-wlglx\") pod \"must-gather-hvnqd\" (UID: \"3686e231-f63d-4889-9fd6-c137e1584028\") " pod="openshift-must-gather-9zq2q/must-gather-hvnqd"
Oct 30 20:07:43.395901 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.395799    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3686e231-f63d-4889-9fd6-c137e1584028-must-gather-output\") pod \"must-gather-hvnqd\" (UID: \"3686e231-f63d-4889-9fd6-c137e1584028\") " pod="openshift-must-gather-9zq2q/must-gather-hvnqd"
Oct 30 20:07:43.396153 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.396135    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3686e231-f63d-4889-9fd6-c137e1584028-must-gather-output\") pod \"must-gather-hvnqd\" (UID: \"3686e231-f63d-4889-9fd6-c137e1584028\") " pod="openshift-must-gather-9zq2q/must-gather-hvnqd"
Oct 30 20:07:43.407172 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.407112    4712 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log" limit=1024
Oct 30 20:07:43.415401 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.415355    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlglx\" (UniqueName: \"kubernetes.io/projected/3686e231-f63d-4889-9fd6-c137e1584028-kube-api-access-wlglx\") pod \"must-gather-hvnqd\" (UID: \"3686e231-f63d-4889-9fd6-c137e1584028\") " pod="openshift-must-gather-9zq2q/must-gather-hvnqd"
Oct 30 20:07:43.467835 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.467810    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zq2q/must-gather-hvnqd"
Oct 30 20:07:43.759620 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.759592    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zq2q/must-gather-hvnqd"]
Oct 30 20:07:43.761084 ip-10-0-140-248 kubenswrapper[4712]: W1030 20:07:43.761056    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3686e231_f63d_4889_9fd6_c137e1584028.slice/crio-19a6e7a8bdc9c0dd838ec74a138ade05d92e0c700e1d4cb52016aa0e278c3a19 WatchSource:0}: Error finding container 19a6e7a8bdc9c0dd838ec74a138ade05d92e0c700e1d4cb52016aa0e278c3a19: Status 404 returned error can't find the container with id 19a6e7a8bdc9c0dd838ec74a138ade05d92e0c700e1d4cb52016aa0e278c3a19
Oct 30 20:07:43.763101 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:43.763084    4712 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:07:44.210887 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:44.210857    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zq2q/must-gather-hvnqd" event={"ID":"3686e231-f63d-4889-9fd6-c137e1584028","Type":"ContainerStarted","Data":"19a6e7a8bdc9c0dd838ec74a138ade05d92e0c700e1d4cb52016aa0e278c3a19"}
Oct 30 20:07:45.229882 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:45.229845    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zq2q/must-gather-hvnqd" event={"ID":"3686e231-f63d-4889-9fd6-c137e1584028","Type":"ContainerStarted","Data":"3ea5e5531b26ce1ce6b9f1080c7225f11f92a0cbc0171a8d8e94dc8fefa2c9e3"}
Oct 30 20:07:45.230313 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:45.229892    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zq2q/must-gather-hvnqd" event={"ID":"3686e231-f63d-4889-9fd6-c137e1584028","Type":"ContainerStarted","Data":"7c0c845f4b2dc62ef684c57a7df3b31c7b7d639135d05ad84ea97d807368bf28"}
Oct 30 20:07:45.275658 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:45.275608    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9zq2q/must-gather-hvnqd" podStartSLOduration=1.550829722 podStartE2EDuration="2.275595521s" totalImagesPullingTime="724.765799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:45.273476002 +0000 UTC m=+2818.572738130" watchObservedRunningTime="2026-10-30 20:07:45.275595521 +0000 UTC m=+2818.574857650"
Oct 30 20:07:47.497514 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:47.497485    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-27xsq_bbd278f9-76f5-41ed-93d1-435804741834/global-pull-secret-syncer/0.log"
Oct 30 20:07:48.413296 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:48.410210    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l65sf_954bf0fe-4287-4435-9abb-fd06dc478099/konnectivity-agent/0.log"
Oct 30 20:07:48.998305 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:48.998261    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-248.ec2.internal_4e9154b2294c90c44348210589c7863a/haproxy/0.log"
Oct 30 20:07:55.259384 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:55.256081    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-fgcgf_f055912e-ab5f-4098-ba65-bc664b7aee30/kuadrant-console-plugin/0.log"
Oct 30 20:07:57.282801 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.282748    4712 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"]
Oct 30 20:07:57.286115 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.286095    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:07:57.305693 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.305654    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"]
Oct 30 20:07:57.459023 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.458985    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqjtj\" (UniqueName: \"kubernetes.io/projected/2720453d-5db6-4667-9b87-85436e3a4f7b-kube-api-access-rqjtj\") pod \"perf-node-gather-daemonset-kp77s\" (UID: \"2720453d-5db6-4667-9b87-85436e3a4f7b\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:07:57.459207 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.459038    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2720453d-5db6-4667-9b87-85436e3a4f7b-lib-modules\") pod \"perf-node-gather-daemonset-kp77s\" (UID: \"2720453d-5db6-4667-9b87-85436e3a4f7b\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:07:57.459207 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.459174    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2720453d-5db6-4667-9b87-85436e3a4f7b-proc\") pod \"perf-node-gather-daemonset-kp77s\" (UID: \"2720453d-5db6-4667-9b87-85436e3a4f7b\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:07:57.459317 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.459233    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2720453d-5db6-4667-9b87-85436e3a4f7b-sys\") pod \"perf-node-gather-daemonset-kp77s\" (UID: \"2720453d-5db6-4667-9b87-85436e3a4f7b\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:07:57.459317 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.459270    4712 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2720453d-5db6-4667-9b87-85436e3a4f7b-podres\") pod \"perf-node-gather-daemonset-kp77s\" (UID: \"2720453d-5db6-4667-9b87-85436e3a4f7b\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:07:57.560259 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.560183    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqjtj\" (UniqueName: \"kubernetes.io/projected/2720453d-5db6-4667-9b87-85436e3a4f7b-kube-api-access-rqjtj\") pod \"perf-node-gather-daemonset-kp77s\" (UID: \"2720453d-5db6-4667-9b87-85436e3a4f7b\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:07:57.560259 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.560221    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2720453d-5db6-4667-9b87-85436e3a4f7b-lib-modules\") pod \"perf-node-gather-daemonset-kp77s\" (UID: \"2720453d-5db6-4667-9b87-85436e3a4f7b\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:07:57.560259 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.560243    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2720453d-5db6-4667-9b87-85436e3a4f7b-proc\") pod \"perf-node-gather-daemonset-kp77s\" (UID: \"2720453d-5db6-4667-9b87-85436e3a4f7b\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:07:57.560571 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.560279    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2720453d-5db6-4667-9b87-85436e3a4f7b-sys\") pod \"perf-node-gather-daemonset-kp77s\" (UID: \"2720453d-5db6-4667-9b87-85436e3a4f7b\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:07:57.560571 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.560304    4712 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2720453d-5db6-4667-9b87-85436e3a4f7b-podres\") pod \"perf-node-gather-daemonset-kp77s\" (UID: \"2720453d-5db6-4667-9b87-85436e3a4f7b\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:07:57.560571 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.560344    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2720453d-5db6-4667-9b87-85436e3a4f7b-proc\") pod \"perf-node-gather-daemonset-kp77s\" (UID: \"2720453d-5db6-4667-9b87-85436e3a4f7b\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:07:57.560571 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.560353    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2720453d-5db6-4667-9b87-85436e3a4f7b-lib-modules\") pod \"perf-node-gather-daemonset-kp77s\" (UID: \"2720453d-5db6-4667-9b87-85436e3a4f7b\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:07:57.560571 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.560441    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2720453d-5db6-4667-9b87-85436e3a4f7b-podres\") pod \"perf-node-gather-daemonset-kp77s\" (UID: \"2720453d-5db6-4667-9b87-85436e3a4f7b\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:07:57.560571 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.560452    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2720453d-5db6-4667-9b87-85436e3a4f7b-sys\") pod \"perf-node-gather-daemonset-kp77s\" (UID: \"2720453d-5db6-4667-9b87-85436e3a4f7b\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:07:57.580741 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.580708    4712 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqjtj\" (UniqueName: \"kubernetes.io/projected/2720453d-5db6-4667-9b87-85436e3a4f7b-kube-api-access-rqjtj\") pod \"perf-node-gather-daemonset-kp77s\" (UID: \"2720453d-5db6-4667-9b87-85436e3a4f7b\") " pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:07:57.607212 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.606515    4712 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:07:57.914653 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:57.914629    4712 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"]
Oct 30 20:07:57.916056 ip-10-0-140-248 kubenswrapper[4712]: W1030 20:07:57.916026    4712 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2720453d_5db6_4667_9b87_85436e3a4f7b.slice/crio-2f2d9ca88c5015b2ad60b3da8b5a0d3d504f07454a223094964e3b957a7a9942 WatchSource:0}: Error finding container 2f2d9ca88c5015b2ad60b3da8b5a0d3d504f07454a223094964e3b957a7a9942: Status 404 returned error can't find the container with id 2f2d9ca88c5015b2ad60b3da8b5a0d3d504f07454a223094964e3b957a7a9942
Oct 30 20:07:58.341596 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:58.341518    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s" event={"ID":"2720453d-5db6-4667-9b87-85436e3a4f7b","Type":"ContainerStarted","Data":"39075b5d53bcffb517951fe4b705952c3ae59fc1656efae1b98aa5df85d97196"}
Oct 30 20:07:58.341596 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:58.341568    4712 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s" event={"ID":"2720453d-5db6-4667-9b87-85436e3a4f7b","Type":"ContainerStarted","Data":"2f2d9ca88c5015b2ad60b3da8b5a0d3d504f07454a223094964e3b957a7a9942"}
Oct 30 20:07:58.342039 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:58.341671    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:07:58.597620 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:58.597546    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7tqb7_5bfa2f12-7480-4a2c-9373-77087aea1a3c/node-exporter/0.log"
Oct 30 20:07:58.710563 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:58.710530    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7tqb7_5bfa2f12-7480-4a2c-9373-77087aea1a3c/kube-rbac-proxy/0.log"
Oct 30 20:07:58.830622 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:07:58.830556    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7tqb7_5bfa2f12-7480-4a2c-9373-77087aea1a3c/init-textfile/0.log"
Oct 30 20:08:00.642877 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:00.642844    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-wbc4b_993b76e2-e387-4307-9309-96a7073b69cf/prometheus-operator/0.log"
Oct 30 20:08:00.761381 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:00.760865    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-wbc4b_993b76e2-e387-4307-9309-96a7073b69cf/kube-rbac-proxy/0.log"
Oct 30 20:08:00.872636 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:00.872603    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-r8gv2_a6076fa2-677c-4d8e-871c-238ec152066c/prometheus-operator-admission-webhook/0.log"
Oct 30 20:08:04.363662 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:04.363632    4712 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s"
Oct 30 20:08:04.409656 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:04.409608    4712 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9zq2q/perf-node-gather-daemonset-kp77s" podStartSLOduration=7.409593039 podStartE2EDuration="7.409593039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:58.385304977 +0000 UTC m=+2831.684567108" watchObservedRunningTime="2026-10-30 20:08:04.409593039 +0000 UTC m=+2837.708855196"
Oct 30 20:08:05.152238 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:05.152199    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-hnr6l_52a21ddd-696e-4d75-a0e0-424aba32607d/download-server/0.log"
Oct 30 20:08:06.393075 ip-10-0-140-248 kubenswrapper[4712]: E1030 20:08:06.391291    4712 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dxp6q" containerName="extractor"
Oct 30 20:08:08.703104 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:08.703072    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bm87v_1b368ff6-93f5-472f-97ae-3f9ffee9102d/dns/0.log"
Oct 30 20:08:08.814872 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:08.814848    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bm87v_1b368ff6-93f5-472f-97ae-3f9ffee9102d/kube-rbac-proxy/0.log"
Oct 30 20:08:09.346699 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:09.345345    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n9frl_90c7079a-e03a-4d3e-bb41-b61ce36292ec/dns-node-resolver/0.log"
Oct 30 20:08:10.065584 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:10.065550    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58487bf9b4-lck4n_cd1c6ba9-e503-4755-89c2-1700e80c4e12/kube-auth-proxy/0.log"
Oct 30 20:08:10.133384 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:10.133340    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-28vt6_8cbf06ec-e967-4cc4-9d06-6629e3be62da/istio-proxy/0.log"
Oct 30 20:08:10.383439 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:10.383406    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-cdt2t_f1c940f3-a6ba-45e5-bb26-2997810c97b6/istio-proxy/0.log"
Oct 30 20:08:10.391006 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:10.390983    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7974c9c7fd-g85kb_bcc2819e-b758-4cc7-b55b-b5fee59a83ff/registry/0.log"
Oct 30 20:08:10.546373 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:10.546343    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-rflrd_75fe15f5-086d-455e-af75-a99955780e72/istio-proxy/0.log"
Oct 30 20:08:10.601394 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:10.601354    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qrsz7_a3e454ca-29fa-4b1d-8c8d-5db1c6c6cada/node-ca/0.log"
Oct 30 20:08:10.799872 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:10.799803    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-7p7sg_17538073-ba9e-44e4-98b1-80a78223f37d/payload-pre-processing/0.log"
Oct 30 20:08:13.571281 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:13.571234    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58487bf9b4-lck4n_cd1c6ba9-e503-4755-89c2-1700e80c4e12/kube-auth-proxy/0.log"
Oct 30 20:08:13.694578 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:13.694551    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-28vt6_8cbf06ec-e967-4cc4-9d06-6629e3be62da/istio-proxy/0.log"
Oct 30 20:08:14.208207 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:14.208157    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58487bf9b4-lck4n_cd1c6ba9-e503-4755-89c2-1700e80c4e12/kube-auth-proxy/0.log"
Oct 30 20:08:14.230812 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:14.230788    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-cdt2t_f1c940f3-a6ba-45e5-bb26-2997810c97b6/istio-proxy/0.log"
Oct 30 20:08:14.275935 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:14.275914    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-28vt6_8cbf06ec-e967-4cc4-9d06-6629e3be62da/istio-proxy/0.log"
Oct 30 20:08:14.533729 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:14.533603    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-cdt2t_f1c940f3-a6ba-45e5-bb26-2997810c97b6/istio-proxy/0.log"
Oct 30 20:08:14.543313 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:14.542771    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-rflrd_75fe15f5-086d-455e-af75-a99955780e72/istio-proxy/0.log"
Oct 30 20:08:14.701456 ip-10-0-140-248 kubenswrapper[4712]: I1030 20:08:14.701430    4712 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-rflrd_75fe15f5-086d-455e-af75-a99955780e72/istio-proxy/0.log"